Note: This is test shard 2 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from RebalanceIgnoredTserversTest
[ RUN ] RebalanceIgnoredTserversTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 08:16:32.452842 21308 test_util.cc:268] Using random seed: 948543682
I20240502 08:16:32.463820 21308 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:16:32.463896 21308 ts_itest-base.cc:113] --------------
I20240502 08:16:32.463944 21308 ts_itest-base.cc:114] 5 tablet servers
I20240502 08:16:32.464007 21308 ts_itest-base.cc:115] 3 replicas per TS
I20240502 08:16:32.464077 21308 ts_itest-base.cc:116] --------------
2024-05-02T08:16:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:16:32Z Disabled control of system clock
I20240502 08:16:32.486312 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.207.62:36729
--webserver_interface=127.20.207.62
--webserver_port=0
--builtin_ntp_servers=127.20.207.20:34501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.207.62:36729
--enable_range_replica_placement=false with env {}
W20240502 08:16:32.791918 21320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:32.792204 21320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:32.792291 21320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:32.799340 21320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:16:32.799434 21320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:32.799508 21320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:16:32.799588 21320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:16:32.807967 21320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:34501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.207.62:36729
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.207.62:36729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.207.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:32.809173 21320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:32.810854 21320 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:32.822103 21325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:32.822973 21326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:32.823647 21328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:32.824226 21320 server_base.cc:1034] running on GCE node
I20240502 08:16:32.824946 21320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:32.826256 21320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:32.827479 21320 hybrid_clock.cc:648] HybridClock initialized: now 1714637792827421 us; error 39 us; skew 500 ppm
I20240502 08:16:32.827790 21320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:32.834969 21320 webserver.cc:460] Webserver started at http://127.20.207.62:37523/ using document root <none> and password file <none>
I20240502 08:16:32.835659 21320 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:32.835790 21320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:32.836097 21320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:32.838568 21320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "aa670ebce13c45e6a650ee45c6213523"
format_stamp: "Formatted at 2024-05-02 08:16:32 on dist-test-slave-169q"
I20240502 08:16:32.839375 21320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "aa670ebce13c45e6a650ee45c6213523"
format_stamp: "Formatted at 2024-05-02 08:16:32 on dist-test-slave-169q"
I20240502 08:16:32.844089 21320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:16:32.847020 21334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:32.848145 21320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:16:32.848295 21320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
uuid: "aa670ebce13c45e6a650ee45c6213523"
format_stamp: "Formatted at 2024-05-02 08:16:32 on dist-test-slave-169q"
I20240502 08:16:32.848455 21320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:32.867631 21320 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:32.868352 21320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:32.868642 21320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:32.885355 21320 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.62:36729
I20240502 08:16:32.885380 21386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.62:36729 every 8 connection(s)
I20240502 08:16:32.886741 21320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:16:32.890352 21387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:32.890352 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 21320
I20240502 08:16:32.890576 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:16:32.899552 21387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523: Bootstrap starting.
I20240502 08:16:32.902240 21387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:32.903132 21387 log.cc:824] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:32.905643 21387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523: No bootstrap required, opened a new log
I20240502 08:16:32.913502 21387 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa670ebce13c45e6a650ee45c6213523" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 36729 } }
I20240502 08:16:32.913854 21387 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:32.913973 21387 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa670ebce13c45e6a650ee45c6213523, State: Initialized, Role: FOLLOWER
I20240502 08:16:32.914747 21387 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa670ebce13c45e6a650ee45c6213523" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 36729 } }
I20240502 08:16:32.915012 21387 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:32.915180 21387 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:32.915377 21387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:32.918334 21387 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa670ebce13c45e6a650ee45c6213523" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 36729 } }
I20240502 08:16:32.918869 21387 leader_election.cc:302] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa670ebce13c45e6a650ee45c6213523; no voters:
I20240502 08:16:32.919454 21387 leader_election.cc:288] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:32.919564 21392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:32.920167 21392 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523 [term 1 LEADER]: Becoming Leader. State: Replica: aa670ebce13c45e6a650ee45c6213523, State: Running, Role: LEADER
I20240502 08:16:32.920711 21392 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa670ebce13c45e6a650ee45c6213523" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 36729 } }
I20240502 08:16:32.921250 21387 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:32.923501 21393 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa670ebce13c45e6a650ee45c6213523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa670ebce13c45e6a650ee45c6213523" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 36729 } } }
I20240502 08:16:32.923913 21393 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:32.926491 21400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:16:32.926470 21394 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa670ebce13c45e6a650ee45c6213523. Latest consensus state: current_term: 1 leader_uuid: "aa670ebce13c45e6a650ee45c6213523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa670ebce13c45e6a650ee45c6213523" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 36729 } } }
I20240502 08:16:32.926841 21394 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:32.931602 21400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:16:32.940296 21400 catalog_manager.cc:1347] Generated new cluster ID: be248ee35ddd4031bf5a26cdba9bd298
I20240502 08:16:32.940407 21400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:16:32.957828 21400 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:16:32.958941 21400 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:16:32.973943 21400 catalog_manager.cc:5857] T 00000000000000000000000000000000 P aa670ebce13c45e6a650ee45c6213523: Generated new TSK 0
I20240502 08:16:32.974773 21400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:16:32.987721 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.1:0
--local_ip_for_outbound_sockets=127.20.207.1
--webserver_interface=127.20.207.1
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36729
--builtin_ntp_servers=127.20.207.20:34501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240502 08:16:33.326596 21412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:33.326869 21412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:33.326959 21412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:33.334218 21412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:33.334436 21412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.1
I20240502 08:16:33.343144 21412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:34501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.207.1
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36729
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.1
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:33.344409 21412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:33.346114 21412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:33.358054 21418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:33.358686 21420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:33.359347 21417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:33.360337 21412 server_base.cc:1034] running on GCE node
I20240502 08:16:33.360906 21412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:33.361766 21412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:33.362959 21412 hybrid_clock.cc:648] HybridClock initialized: now 1714637793362881 us; error 62 us; skew 500 ppm
I20240502 08:16:33.363229 21412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:33.365311 21412 webserver.cc:460] Webserver started at http://127.20.207.1:35025/ using document root <none> and password file <none>
I20240502 08:16:33.366086 21412 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:33.366217 21412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:33.366503 21412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:33.368930 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a52a81badfae455480a1f2efc2663cde"
format_stamp: "Formatted at 2024-05-02 08:16:33 on dist-test-slave-169q"
I20240502 08:16:33.369663 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a52a81badfae455480a1f2efc2663cde"
format_stamp: "Formatted at 2024-05-02 08:16:33 on dist-test-slave-169q"
I20240502 08:16:33.374487 21412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20240502 08:16:33.377517 21427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:33.378729 21412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:16:33.378875 21412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a52a81badfae455480a1f2efc2663cde"
format_stamp: "Formatted at 2024-05-02 08:16:33 on dist-test-slave-169q"
I20240502 08:16:33.379062 21412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:33.432389 21412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:33.433238 21412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:33.433557 21412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:33.434535 21412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:33.436002 21412 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:33.436081 21412 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:33.436209 21412 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:33.436264 21412 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:33.466331 21412 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.1:43449
I20240502 08:16:33.466365 21539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.1:43449 every 8 connection(s)
I20240502 08:16:33.467541 21412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:16:33.476851 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 21412
I20240502 08:16:33.477077 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:16:33.479501 21540 heartbeater.cc:347] Connected to a master server at 127.20.207.62:36729
I20240502 08:16:33.479799 21540 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:33.480365 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.2:0
--local_ip_for_outbound_sockets=127.20.207.2
--webserver_interface=127.20.207.2
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36729
--builtin_ntp_servers=127.20.207.20:34501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 08:16:33.480751 21540 heartbeater.cc:511] Master 127.20.207.62:36729 requested a full tablet report, sending...
I20240502 08:16:33.482820 21352 ts_manager.cc:194] Registered new tserver with Master: a52a81badfae455480a1f2efc2663cde (127.20.207.1:43449)
I20240502 08:16:33.484350 21352 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.1:40857
W20240502 08:16:33.793687 21544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:33.794016 21544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:33.794179 21544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:33.801502 21544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:33.801717 21544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.2
I20240502 08:16:33.810302 21544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:34501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.207.2
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36729
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.2
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:33.811661 21544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:33.813354 21544 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:33.825372 21551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:33.826272 21544 server_base.cc:1034] running on GCE node
W20240502 08:16:33.826983 21550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:33.827536 21553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:33.828193 21544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:33.829090 21544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:33.830298 21544 hybrid_clock.cc:648] HybridClock initialized: now 1714637793830232 us; error 37 us; skew 500 ppm
I20240502 08:16:33.830598 21544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:33.832852 21544 webserver.cc:460] Webserver started at http://127.20.207.2:38373/ using document root <none> and password file <none>
I20240502 08:16:33.833550 21544 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:33.833667 21544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:33.833962 21544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:33.836603 21544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "dff24ad45bac4b70a8c801a8de5188c5"
format_stamp: "Formatted at 2024-05-02 08:16:33 on dist-test-slave-169q"
I20240502 08:16:33.837289 21544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "dff24ad45bac4b70a8c801a8de5188c5"
format_stamp: "Formatted at 2024-05-02 08:16:33 on dist-test-slave-169q"
I20240502 08:16:33.842227 21544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 08:16:33.845409 21559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:33.846807 21544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:33.846956 21544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "dff24ad45bac4b70a8c801a8de5188c5"
format_stamp: "Formatted at 2024-05-02 08:16:33 on dist-test-slave-169q"
I20240502 08:16:33.847121 21544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:33.866309 21544 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:33.867125 21544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:33.867370 21544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:33.868250 21544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:33.869648 21544 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:33.869719 21544 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:33.869803 21544 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:33.869845 21544 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:33.902576 21544 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.2:46145
I20240502 08:16:33.902606 21671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.2:46145 every 8 connection(s)
I20240502 08:16:33.903847 21544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:16:33.907599 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 21544
I20240502 08:16:33.907785 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:16:33.911123 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.3:0
--local_ip_for_outbound_sockets=127.20.207.3
--webserver_interface=127.20.207.3
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36729
--builtin_ntp_servers=127.20.207.20:34501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 08:16:33.920362 21672 heartbeater.cc:347] Connected to a master server at 127.20.207.62:36729
I20240502 08:16:33.920684 21672 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:33.921567 21672 heartbeater.cc:511] Master 127.20.207.62:36729 requested a full tablet report, sending...
I20240502 08:16:33.923234 21352 ts_manager.cc:194] Registered new tserver with Master: dff24ad45bac4b70a8c801a8de5188c5 (127.20.207.2:46145)
I20240502 08:16:33.924495 21352 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.2:39701
W20240502 08:16:34.232666 21675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:34.232972 21675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:34.233068 21675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:34.240558 21675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:34.240741 21675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.3
I20240502 08:16:34.249554 21675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:34501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.207.3
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36729
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.3
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:34.250869 21675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:34.252732 21675 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:34.264823 21681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:34.265636 21683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:34.266450 21680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:34.267172 21675 server_base.cc:1034] running on GCE node
I20240502 08:16:34.267807 21675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:34.268718 21675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:34.269933 21675 hybrid_clock.cc:648] HybridClock initialized: now 1714637794269873 us; error 48 us; skew 500 ppm
I20240502 08:16:34.270246 21675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:34.272586 21675 webserver.cc:460] Webserver started at http://127.20.207.3:35497/ using document root <none> and password file <none>
I20240502 08:16:34.273298 21675 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:34.273425 21675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:34.273734 21675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:34.276353 21675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0cf829b59bc94e65aa57acb8ca9fb29c"
format_stamp: "Formatted at 2024-05-02 08:16:34 on dist-test-slave-169q"
I20240502 08:16:34.277117 21675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0cf829b59bc94e65aa57acb8ca9fb29c"
format_stamp: "Formatted at 2024-05-02 08:16:34 on dist-test-slave-169q"
I20240502 08:16:34.282438 21675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240502 08:16:34.286065 21689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:34.287483 21675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:16:34.287657 21675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0cf829b59bc94e65aa57acb8ca9fb29c"
format_stamp: "Formatted at 2024-05-02 08:16:34 on dist-test-slave-169q"
I20240502 08:16:34.287835 21675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:34.326488 21675 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:34.327490 21675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:34.327783 21675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:34.328784 21675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:34.330300 21675 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:34.330382 21675 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:34.330516 21675 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:34.330572 21675 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:34.362509 21675 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.3:37659
I20240502 08:16:34.362552 21802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.3:37659 every 8 connection(s)
I20240502 08:16:34.363837 21675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:16:34.371618 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 21675
I20240502 08:16:34.371886 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:16:34.375200 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.4:0
--local_ip_for_outbound_sockets=127.20.207.4
--webserver_interface=127.20.207.4
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36729
--builtin_ntp_servers=127.20.207.20:34501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 08:16:34.377202 21803 heartbeater.cc:347] Connected to a master server at 127.20.207.62:36729
I20240502 08:16:34.377498 21803 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:34.378371 21803 heartbeater.cc:511] Master 127.20.207.62:36729 requested a full tablet report, sending...
I20240502 08:16:34.379778 21352 ts_manager.cc:194] Registered new tserver with Master: 0cf829b59bc94e65aa57acb8ca9fb29c (127.20.207.3:37659)
I20240502 08:16:34.380681 21352 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.3:46965
I20240502 08:16:34.486940 21540 heartbeater.cc:503] Master 127.20.207.62:36729 was elected leader, sending a full tablet report...
W20240502 08:16:34.694763 21806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:34.695082 21806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:34.695226 21806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:34.702798 21806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:34.703035 21806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.4
I20240502 08:16:34.711828 21806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:34501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.207.4
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36729
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.4
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:34.713181 21806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:34.714949 21806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:34.726506 21811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:34.727406 21812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:34.728175 21814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:34.728996 21806 server_base.cc:1034] running on GCE node
I20240502 08:16:34.729597 21806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:34.730602 21806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:34.731827 21806 hybrid_clock.cc:648] HybridClock initialized: now 1714637794731769 us; error 39 us; skew 500 ppm
I20240502 08:16:34.732120 21806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:34.734455 21806 webserver.cc:460] Webserver started at http://127.20.207.4:43509/ using document root <none> and password file <none>
I20240502 08:16:34.735183 21806 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:34.735327 21806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:34.735642 21806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:34.738243 21806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "6d3505d99b2749c2bf96fb19d87f7cc9"
format_stamp: "Formatted at 2024-05-02 08:16:34 on dist-test-slave-169q"
I20240502 08:16:34.739027 21806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "6d3505d99b2749c2bf96fb19d87f7cc9"
format_stamp: "Formatted at 2024-05-02 08:16:34 on dist-test-slave-169q"
I20240502 08:16:34.744061 21806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:16:34.747190 21820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:34.748445 21806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:16:34.748610 21806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "6d3505d99b2749c2bf96fb19d87f7cc9"
format_stamp: "Formatted at 2024-05-02 08:16:34 on dist-test-slave-169q"
I20240502 08:16:34.748777 21806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:34.783434 21806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:34.784349 21806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:34.784631 21806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:34.785550 21806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:34.787036 21806 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:34.787117 21806 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:34.787252 21806 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:34.787314 21806 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:34.817584 21806 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.4:39723
I20240502 08:16:34.817633 21933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.4:39723 every 8 connection(s)
I20240502 08:16:34.818887 21806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:16:34.823999 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 21806
I20240502 08:16:34.824146 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:16:34.827034 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.5:0
--local_ip_for_outbound_sockets=127.20.207.5
--webserver_interface=127.20.207.5
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36729
--builtin_ntp_servers=127.20.207.20:34501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 08:16:34.831509 21934 heartbeater.cc:347] Connected to a master server at 127.20.207.62:36729
I20240502 08:16:34.831810 21934 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:34.832540 21934 heartbeater.cc:511] Master 127.20.207.62:36729 requested a full tablet report, sending...
I20240502 08:16:34.833873 21352 ts_manager.cc:194] Registered new tserver with Master: 6d3505d99b2749c2bf96fb19d87f7cc9 (127.20.207.4:39723)
I20240502 08:16:34.834621 21352 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.4:35943
I20240502 08:16:34.927129 21672 heartbeater.cc:503] Master 127.20.207.62:36729 was elected leader, sending a full tablet report...
W20240502 08:16:35.149329 21937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:35.149664 21937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:35.149816 21937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:35.157600 21937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:35.157833 21937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.5
I20240502 08:16:35.166896 21937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:34501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.207.5
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36729
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.5
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:35.168270 21937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:35.170027 21937 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:35.181845 21943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:35.182583 21942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:35.183461 21945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:35.184159 21937 server_base.cc:1034] running on GCE node
I20240502 08:16:35.184752 21937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:35.185726 21937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:35.186959 21937 hybrid_clock.cc:648] HybridClock initialized: now 1714637795186900 us; error 39 us; skew 500 ppm
I20240502 08:16:35.187259 21937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:35.189441 21937 webserver.cc:460] Webserver started at http://127.20.207.5:40491/ using document root <none> and password file <none>
I20240502 08:16:35.190200 21937 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:35.190328 21937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:35.190624 21937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:35.193212 21937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "519ea9904cdf4058987770d94b7a3877"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-169q"
I20240502 08:16:35.193964 21937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "519ea9904cdf4058987770d94b7a3877"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-169q"
I20240502 08:16:35.199168 21937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 08:16:35.202503 21951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.203821 21937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240502 08:16:35.203984 21937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "519ea9904cdf4058987770d94b7a3877"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-169q"
I20240502 08:16:35.204160 21937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:35.230693 21937 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:35.231575 21937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:35.231863 21937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:35.232829 21937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:35.234364 21937 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:35.234445 21937 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.234575 21937 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:35.234632 21937 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.267184 21937 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.5:35139
I20240502 08:16:35.267225 22063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.5:35139 every 8 connection(s)
I20240502 08:16:35.268460 21937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:16:35.273784 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 21937
I20240502 08:16:35.273993 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:16:35.280426 22064 heartbeater.cc:347] Connected to a master server at 127.20.207.62:36729
I20240502 08:16:35.280706 22064 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:35.281451 22064 heartbeater.cc:511] Master 127.20.207.62:36729 requested a full tablet report, sending...
I20240502 08:16:35.282764 21352 ts_manager.cc:194] Registered new tserver with Master: 519ea9904cdf4058987770d94b7a3877 (127.20.207.5:35139)
I20240502 08:16:35.283504 21352 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.5:41639
I20240502 08:16:35.289760 21308 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 08:16:35.309734 21352 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33082:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 08:16:35.339318 21738 tablet_service.cc:1467] Processing CreateTablet for tablet f0fe00b226b44ca698b98e5821dcf0f2 (DEFAULT_TABLE table=TestTable [id=5abb790ca95c4587993a182277e4268b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.340044 21999 tablet_service.cc:1467] Processing CreateTablet for tablet f0fe00b226b44ca698b98e5821dcf0f2 (DEFAULT_TABLE table=TestTable [id=5abb790ca95c4587993a182277e4268b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.340804 21738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0fe00b226b44ca698b98e5821dcf0f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.340358 21869 tablet_service.cc:1467] Processing CreateTablet for tablet f0fe00b226b44ca698b98e5821dcf0f2 (DEFAULT_TABLE table=TestTable [id=5abb790ca95c4587993a182277e4268b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.341457 21999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0fe00b226b44ca698b98e5821dcf0f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.341534 21869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0fe00b226b44ca698b98e5821dcf0f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.350337 22083 tablet_bootstrap.cc:492] T f0fe00b226b44ca698b98e5821dcf0f2 P 0cf829b59bc94e65aa57acb8ca9fb29c: Bootstrap starting.
I20240502 08:16:35.354406 22084 tablet_bootstrap.cc:492] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9: Bootstrap starting.
I20240502 08:16:35.354924 22083 tablet_bootstrap.cc:654] T f0fe00b226b44ca698b98e5821dcf0f2 P 0cf829b59bc94e65aa57acb8ca9fb29c: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.356199 22083 log.cc:824] T f0fe00b226b44ca698b98e5821dcf0f2 P 0cf829b59bc94e65aa57acb8ca9fb29c: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:35.356456 22085 tablet_bootstrap.cc:492] T f0fe00b226b44ca698b98e5821dcf0f2 P 519ea9904cdf4058987770d94b7a3877: Bootstrap starting.
I20240502 08:16:35.358577 22084 tablet_bootstrap.cc:654] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.359759 22085 tablet_bootstrap.cc:654] T f0fe00b226b44ca698b98e5821dcf0f2 P 519ea9904cdf4058987770d94b7a3877: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.360170 22084 log.cc:824] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:35.360750 22085 log.cc:824] T f0fe00b226b44ca698b98e5821dcf0f2 P 519ea9904cdf4058987770d94b7a3877: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:35.363569 22084 tablet_bootstrap.cc:492] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9: No bootstrap required, opened a new log
I20240502 08:16:35.363837 22084 ts_tablet_manager.cc:1367] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 08:16:35.364145 22085 tablet_bootstrap.cc:492] T f0fe00b226b44ca698b98e5821dcf0f2 P 519ea9904cdf4058987770d94b7a3877: No bootstrap required, opened a new log
I20240502 08:16:35.364396 22085 ts_tablet_manager.cc:1367] T f0fe00b226b44ca698b98e5821dcf0f2 P 519ea9904cdf4058987770d94b7a3877: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 08:16:35.364954 22083 tablet_bootstrap.cc:492] T f0fe00b226b44ca698b98e5821dcf0f2 P 0cf829b59bc94e65aa57acb8ca9fb29c: No bootstrap required, opened a new log
I20240502 08:16:35.365199 22083 ts_tablet_manager.cc:1367] T f0fe00b226b44ca698b98e5821dcf0f2 P 0cf829b59bc94e65aa57acb8ca9fb29c: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20240502 08:16:35.372954 22084 raft_consensus.cc:348] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf829b59bc94e65aa57acb8ca9fb29c" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 37659 } } peers { permanent_uuid: "519ea9904cdf4058987770d94b7a3877" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 35139 } } peers { permanent_uuid: "6d3505d99b2749c2bf96fb19d87f7cc9" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 39723 } }
I20240502 08:16:35.373534 22084 raft_consensus.cc:374] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.373708 22084 raft_consensus.cc:729] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d3505d99b2749c2bf96fb19d87f7cc9, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.374157 22085 raft_consensus.cc:348] T f0fe00b226b44ca698b98e5821dcf0f2 P 519ea9904cdf4058987770d94b7a3877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf829b59bc94e65aa57acb8ca9fb29c" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 37659 } } peers { permanent_uuid: "519ea9904cdf4058987770d94b7a3877" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 35139 } } peers { permanent_uuid: "6d3505d99b2749c2bf96fb19d87f7cc9" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 39723 } }
I20240502 08:16:35.374536 22085 raft_consensus.cc:374] T f0fe00b226b44ca698b98e5821dcf0f2 P 519ea9904cdf4058987770d94b7a3877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.374634 22085 raft_consensus.cc:729] T f0fe00b226b44ca698b98e5821dcf0f2 P 519ea9904cdf4058987770d94b7a3877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519ea9904cdf4058987770d94b7a3877, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.374414 22083 raft_consensus.cc:348] T f0fe00b226b44ca698b98e5821dcf0f2 P 0cf829b59bc94e65aa57acb8ca9fb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf829b59bc94e65aa57acb8ca9fb29c" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 37659 } } peers { permanent_uuid: "519ea9904cdf4058987770d94b7a3877" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 35139 } } peers { permanent_uuid: "6d3505d99b2749c2bf96fb19d87f7cc9" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 39723 } }
I20240502 08:16:35.374799 22083 raft_consensus.cc:374] T f0fe00b226b44ca698b98e5821dcf0f2 P 0cf829b59bc94e65aa57acb8ca9fb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.374802 22084 consensus_queue.cc:260] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf829b59bc94e65aa57acb8ca9fb29c" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 37659 } } peers { permanent_uuid: "519ea9904cdf4058987770d94b7a3877" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 35139 } } peers { permanent_uuid: "6d3505d99b2749c2bf96fb19d87f7cc9" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 39723 } }
I20240502 08:16:35.375028 22083 raft_consensus.cc:729] T f0fe00b226b44ca698b98e5821dcf0f2 P 0cf829b59bc94e65aa57acb8ca9fb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf829b59bc94e65aa57acb8ca9fb29c, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.375228 22085 consensus_queue.cc:260] T f0fe00b226b44ca698b98e5821dcf0f2 P 519ea9904cdf4058987770d94b7a3877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf829b59bc94e65aa57acb8ca9fb29c" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 37659 } } peers { permanent_uuid: "519ea9904cdf4058987770d94b7a3877" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 35139 } } peers { permanent_uuid: "6d3505d99b2749c2bf96fb19d87f7cc9" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 39723 } }
I20240502 08:16:35.375831 22083 consensus_queue.cc:260] T f0fe00b226b44ca698b98e5821dcf0f2 P 0cf829b59bc94e65aa57acb8ca9fb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf829b59bc94e65aa57acb8ca9fb29c" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 37659 } } peers { permanent_uuid: "519ea9904cdf4058987770d94b7a3877" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 35139 } } peers { permanent_uuid: "6d3505d99b2749c2bf96fb19d87f7cc9" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 39723 } }
I20240502 08:16:35.380149 21803 heartbeater.cc:503] Master 127.20.207.62:36729 was elected leader, sending a full tablet report...
I20240502 08:16:35.381312 22083 ts_tablet_manager.cc:1398] T f0fe00b226b44ca698b98e5821dcf0f2 P 0cf829b59bc94e65aa57acb8ca9fb29c: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20240502 08:16:35.382256 22064 heartbeater.cc:503] Master 127.20.207.62:36729 was elected leader, sending a full tablet report...
I20240502 08:16:35.383181 22085 ts_tablet_manager.cc:1398] T f0fe00b226b44ca698b98e5821dcf0f2 P 519ea9904cdf4058987770d94b7a3877: Time spent starting tablet: real 0.019s user 0.013s sys 0.005s
I20240502 08:16:35.383347 22084 ts_tablet_manager.cc:1398] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
I20240502 08:16:35.383786 21934 heartbeater.cc:503] Master 127.20.207.62:36729 was elected leader, sending a full tablet report...
W20240502 08:16:35.520133 22065 tablet.cc:2369] T f0fe00b226b44ca698b98e5821dcf0f2 P 519ea9904cdf4058987770d94b7a3877: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:16:35.570847 21935 tablet.cc:2369] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:16:35.616269 21804 tablet.cc:2369] T f0fe00b226b44ca698b98e5821dcf0f2 P 0cf829b59bc94e65aa57acb8ca9fb29c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:35.780542 22089 raft_consensus.cc:482] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:35.780844 22089 raft_consensus.cc:504] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf829b59bc94e65aa57acb8ca9fb29c" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 37659 } } peers { permanent_uuid: "519ea9904cdf4058987770d94b7a3877" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 35139 } } peers { permanent_uuid: "6d3505d99b2749c2bf96fb19d87f7cc9" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 39723 } }
I20240502 08:16:35.782747 22089 leader_election.cc:288] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf829b59bc94e65aa57acb8ca9fb29c (127.20.207.3:37659), 519ea9904cdf4058987770d94b7a3877 (127.20.207.5:35139)
I20240502 08:16:35.789134 21758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fe00b226b44ca698b98e5821dcf0f2" candidate_uuid: "6d3505d99b2749c2bf96fb19d87f7cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf829b59bc94e65aa57acb8ca9fb29c" is_pre_election: true
I20240502 08:16:35.789469 22019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fe00b226b44ca698b98e5821dcf0f2" candidate_uuid: "6d3505d99b2749c2bf96fb19d87f7cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519ea9904cdf4058987770d94b7a3877" is_pre_election: true
I20240502 08:16:35.789746 21758 raft_consensus.cc:2454] T f0fe00b226b44ca698b98e5821dcf0f2 P 0cf829b59bc94e65aa57acb8ca9fb29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d3505d99b2749c2bf96fb19d87f7cc9 in term 0.
I20240502 08:16:35.789917 22019 raft_consensus.cc:2454] T f0fe00b226b44ca698b98e5821dcf0f2 P 519ea9904cdf4058987770d94b7a3877 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d3505d99b2749c2bf96fb19d87f7cc9 in term 0.
I20240502 08:16:35.790395 21821 leader_election.cc:302] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf829b59bc94e65aa57acb8ca9fb29c, 6d3505d99b2749c2bf96fb19d87f7cc9; no voters:
I20240502 08:16:35.790747 22089 raft_consensus.cc:2781] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:35.790875 22089 raft_consensus.cc:482] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:35.790997 22089 raft_consensus.cc:3037] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:35.796057 22089 raft_consensus.cc:504] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf829b59bc94e65aa57acb8ca9fb29c" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 37659 } } peers { permanent_uuid: "519ea9904cdf4058987770d94b7a3877" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 35139 } } peers { permanent_uuid: "6d3505d99b2749c2bf96fb19d87f7cc9" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 39723 } }
I20240502 08:16:35.796720 22089 leader_election.cc:288] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf829b59bc94e65aa57acb8ca9fb29c (127.20.207.3:37659), 519ea9904cdf4058987770d94b7a3877 (127.20.207.5:35139)
I20240502 08:16:35.797204 21758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fe00b226b44ca698b98e5821dcf0f2" candidate_uuid: "6d3505d99b2749c2bf96fb19d87f7cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf829b59bc94e65aa57acb8ca9fb29c"
I20240502 08:16:35.797236 22019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fe00b226b44ca698b98e5821dcf0f2" candidate_uuid: "6d3505d99b2749c2bf96fb19d87f7cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519ea9904cdf4058987770d94b7a3877"
I20240502 08:16:35.797416 22019 raft_consensus.cc:3037] T f0fe00b226b44ca698b98e5821dcf0f2 P 519ea9904cdf4058987770d94b7a3877 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:35.797416 21758 raft_consensus.cc:3037] T f0fe00b226b44ca698b98e5821dcf0f2 P 0cf829b59bc94e65aa57acb8ca9fb29c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:35.800460 22019 raft_consensus.cc:2454] T f0fe00b226b44ca698b98e5821dcf0f2 P 519ea9904cdf4058987770d94b7a3877 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d3505d99b2749c2bf96fb19d87f7cc9 in term 1.
I20240502 08:16:35.800460 21758 raft_consensus.cc:2454] T f0fe00b226b44ca698b98e5821dcf0f2 P 0cf829b59bc94e65aa57acb8ca9fb29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d3505d99b2749c2bf96fb19d87f7cc9 in term 1.
I20240502 08:16:35.800863 21821 leader_election.cc:302] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf829b59bc94e65aa57acb8ca9fb29c, 6d3505d99b2749c2bf96fb19d87f7cc9; no voters:
I20240502 08:16:35.801139 22089 raft_consensus.cc:2781] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:35.801537 22089 raft_consensus.cc:686] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [term 1 LEADER]: Becoming Leader. State: Replica: 6d3505d99b2749c2bf96fb19d87f7cc9, State: Running, Role: LEADER
I20240502 08:16:35.802060 22089 consensus_queue.cc:237] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf829b59bc94e65aa57acb8ca9fb29c" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 37659 } } peers { permanent_uuid: "519ea9904cdf4058987770d94b7a3877" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 35139 } } peers { permanent_uuid: "6d3505d99b2749c2bf96fb19d87f7cc9" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 39723 } }
I20240502 08:16:35.806586 21352 catalog_manager.cc:5484] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d3505d99b2749c2bf96fb19d87f7cc9 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "6d3505d99b2749c2bf96fb19d87f7cc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf829b59bc94e65aa57acb8ca9fb29c" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "519ea9904cdf4058987770d94b7a3877" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d3505d99b2749c2bf96fb19d87f7cc9" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 39723 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:35.899353 21308 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 08:16:35.901839 21308 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a52a81badfae455480a1f2efc2663cde to finish bootstrapping
I20240502 08:16:35.911434 21308 ts_itest-base.cc:233] Waiting for 0 tablets on tserver dff24ad45bac4b70a8c801a8de5188c5 to finish bootstrapping
I20240502 08:16:35.930248 21308 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0cf829b59bc94e65aa57acb8ca9fb29c to finish bootstrapping
I20240502 08:16:35.937673 21308 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6d3505d99b2749c2bf96fb19d87f7cc9 to finish bootstrapping
I20240502 08:16:35.944913 21308 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 519ea9904cdf4058987770d94b7a3877 to finish bootstrapping
I20240502 08:16:36.336328 22112 consensus_queue.cc:1035] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519ea9904cdf4058987770d94b7a3877" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:36.345708 22115 consensus_queue.cc:1035] T f0fe00b226b44ca698b98e5821dcf0f2 P 6d3505d99b2749c2bf96fb19d87f7cc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf829b59bc94e65aa57acb8ca9fb29c" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:36.411706 21738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:36.416790 21999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:36.419692 21869 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:36.421931 21475 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:36.440964 21607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:37.022500 21738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:37.022646 21999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:37.027419 21869 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:37.039391 21475 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:37.039410 21607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:37.192658 21738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:37.196409 21999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:37.197453 21869 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:37.202368 21607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:37.209636 21475 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:37.391225 21738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:37.392120 21999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:37.398767 21607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:37.399791 21475 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:37.406523 21869 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:37.944578 21350 ts_manager.cc:292] Set tserver state for a52a81badfae455480a1f2efc2663cde to MAINTENANCE_MODE
I20240502 08:16:39.394235 21738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:39.400365 21999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:39.407851 21475 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:39.410630 21869 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:39.428068 21607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:39.574378 21869 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:39.574626 21999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:39.575062 21738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:39.576148 21475 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:39.592334 21607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:39.744854 21738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:39.746362 21869 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:39.748196 21999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:39.751178 21607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:39.763377 21475 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:39.909893 21738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:39.922255 21999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:39.931777 21475 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:39.933395 21869 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:39.962473 21607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:40.100651 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 21544
I20240502 08:16:40.102206 21667 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:40.202982 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 21544
I20240502 08:16:40.701550 21738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:40.702255 21869 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:40.704236 21475 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:40.705256 21999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:40.826686 21738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:40.835314 21999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:40.839219 21869 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:40.842752 21475 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:40.994102 21738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:41.000432 21475 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:41.002039 21999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:41.009038 21869 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:41.135349 21738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:41.139747 21999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:41.150261 21475 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:41.158808 21869 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:41.297410 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 21412
I20240502 08:16:41.299130 21535 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:41.407572 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 21412
I20240502 08:16:41.426947 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 21675
I20240502 08:16:41.427995 21797 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:41.535631 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 21675
I20240502 08:16:41.555335 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 21806
I20240502 08:16:41.556428 21928 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:41.665503 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 21806
I20240502 08:16:41.685279 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 21937
I20240502 08:16:41.686408 22059 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:41.793007 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 21937
I20240502 08:16:41.812852 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 21320
I20240502 08:16:41.814100 21382 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:41.945360 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 21320
2024-05-02T08:16:41Z chronyd exiting
[ OK ] RebalanceIgnoredTserversTest.Basic (9540 ms)
[----------] 1 test from RebalanceIgnoredTserversTest (9540 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1
I20240502 08:16:41.990970 21308 test_util.cc:268] Using random seed: 958081801
I20240502 08:16:41.992949 21308 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:16:41.993036 21308 ts_itest-base.cc:113] --------------
I20240502 08:16:41.993085 21308 ts_itest-base.cc:114] 5 tablet servers
I20240502 08:16:41.993163 21308 ts_itest-base.cc:115] 3 replicas per TS
I20240502 08:16:41.993212 21308 ts_itest-base.cc:116] --------------
2024-05-02T08:16:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:16:42Z Disabled control of system clock
I20240502 08:16:42.017766 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.207.62:41817
--webserver_interface=127.20.207.62
--webserver_port=0
--builtin_ntp_servers=127.20.207.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.207.62:41817
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 08:16:42.352075 22536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:42.352437 22536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:42.352597 22536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:42.360173 22536 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:42.360352 22536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:16:42.360452 22536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:42.360543 22536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:16:42.360612 22536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:16:42.370040 22536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.207.62:41817
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.207.62:41817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.207.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:42.371466 22536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:42.373226 22536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:42.384963 22542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:42.385619 22544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:42.386261 22541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:42.386837 22536 server_base.cc:1034] running on GCE node
I20240502 08:16:42.387673 22536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:42.389127 22536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:42.390352 22536 hybrid_clock.cc:648] HybridClock initialized: now 1714637802390303 us; error 46 us; skew 500 ppm
I20240502 08:16:42.390717 22536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:42.393169 22536 webserver.cc:460] Webserver started at http://127.20.207.62:41089/ using document root <none> and password file <none>
I20240502 08:16:42.393913 22536 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:42.394058 22536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:42.394412 22536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:42.397152 22536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5650648934394bf8839bcdfff1d4dd67"
format_stamp: "Formatted at 2024-05-02 08:16:42 on dist-test-slave-169q"
I20240502 08:16:42.397930 22536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5650648934394bf8839bcdfff1d4dd67"
format_stamp: "Formatted at 2024-05-02 08:16:42 on dist-test-slave-169q"
I20240502 08:16:42.403425 22536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 08:16:42.406919 22550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:42.408320 22536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240502 08:16:42.408501 22536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5650648934394bf8839bcdfff1d4dd67"
format_stamp: "Formatted at 2024-05-02 08:16:42 on dist-test-slave-169q"
I20240502 08:16:42.408720 22536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:42.424665 22536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:42.425561 22536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:42.425855 22536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:42.443688 22536 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.62:41817
I20240502 08:16:42.443709 22602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.62:41817 every 8 connection(s)
I20240502 08:16:42.445062 22536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:16:42.448910 22603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.455236 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 22536
I20240502 08:16:42.455482 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:16:42.459522 22603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67: Bootstrap starting.
I20240502 08:16:42.462656 22603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.463761 22603 log.cc:824] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:42.466467 22603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67: No bootstrap required, opened a new log
I20240502 08:16:42.473663 22603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5650648934394bf8839bcdfff1d4dd67" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 41817 } }
I20240502 08:16:42.474040 22603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.474155 22603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5650648934394bf8839bcdfff1d4dd67, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.474751 22603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5650648934394bf8839bcdfff1d4dd67" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 41817 } }
I20240502 08:16:42.474953 22603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:42.475076 22603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:42.475224 22603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:42.477263 22603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5650648934394bf8839bcdfff1d4dd67" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 41817 } }
I20240502 08:16:42.477830 22603 leader_election.cc:302] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5650648934394bf8839bcdfff1d4dd67; no voters:
I20240502 08:16:42.478497 22603 leader_election.cc:288] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:42.478646 22608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:42.479331 22608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67 [term 1 LEADER]: Becoming Leader. State: Replica: 5650648934394bf8839bcdfff1d4dd67, State: Running, Role: LEADER
I20240502 08:16:42.479876 22608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5650648934394bf8839bcdfff1d4dd67" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 41817 } }
I20240502 08:16:42.480551 22603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:42.482710 22609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5650648934394bf8839bcdfff1d4dd67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5650648934394bf8839bcdfff1d4dd67" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 41817 } } }
I20240502 08:16:42.483129 22609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:42.483847 22610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5650648934394bf8839bcdfff1d4dd67. Latest consensus state: current_term: 1 leader_uuid: "5650648934394bf8839bcdfff1d4dd67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5650648934394bf8839bcdfff1d4dd67" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 41817 } } }
I20240502 08:16:42.484298 22610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:42.484935 22616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:16:42.491322 22616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:16:42.500743 22616 catalog_manager.cc:1347] Generated new cluster ID: 9ab1a06cc6084bdab71dc1e0ddcedcc9
I20240502 08:16:42.500871 22616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:16:42.508697 22616 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:16:42.510450 22616 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:16:42.519222 22616 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5650648934394bf8839bcdfff1d4dd67: Generated new TSK 0
I20240502 08:16:42.520339 22616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:16:42.533089 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.1:0
--local_ip_for_outbound_sockets=127.20.207.1
--webserver_interface=127.20.207.1
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41817
--builtin_ntp_servers=127.20.207.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:16:42.859051 22627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:42.859333 22627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:42.859470 22627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:42.867005 22627 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:42.867151 22627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:42.867333 22627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.1
I20240502 08:16:42.878392 22627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.207.1
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41817
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.1
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:42.881426 22627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:42.883172 22627 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:42.895582 22632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:42.896034 22635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:42.896631 22633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:42.897295 22627 server_base.cc:1034] running on GCE node
I20240502 08:16:42.897869 22627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:42.898805 22627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:42.900005 22627 hybrid_clock.cc:648] HybridClock initialized: now 1714637802899964 us; error 48 us; skew 500 ppm
I20240502 08:16:42.900321 22627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:42.902591 22627 webserver.cc:460] Webserver started at http://127.20.207.1:37389/ using document root <none> and password file <none>
I20240502 08:16:42.903288 22627 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:42.903412 22627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:42.903730 22627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:42.906323 22627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "00844c1f5dba4c0bbe65483833cadf96"
format_stamp: "Formatted at 2024-05-02 08:16:42 on dist-test-slave-169q"
I20240502 08:16:42.907070 22627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "00844c1f5dba4c0bbe65483833cadf96"
format_stamp: "Formatted at 2024-05-02 08:16:42 on dist-test-slave-169q"
I20240502 08:16:42.912052 22627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:16:42.915369 22641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:42.916664 22627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:16:42.916815 22627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "00844c1f5dba4c0bbe65483833cadf96"
format_stamp: "Formatted at 2024-05-02 08:16:42 on dist-test-slave-169q"
I20240502 08:16:42.917012 22627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:42.942694 22627 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:42.943578 22627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:42.943850 22627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:42.944825 22627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:42.946369 22627 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:42.946468 22627 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:42.946581 22627 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:42.946653 22627 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:42.976980 22627 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.1:36851
I20240502 08:16:42.977028 22754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.1:36851 every 8 connection(s)
I20240502 08:16:42.978289 22627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:16:42.981083 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 22627
I20240502 08:16:42.981285 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:16:42.984930 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.2:0
--local_ip_for_outbound_sockets=127.20.207.2
--webserver_interface=127.20.207.2
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41817
--builtin_ntp_servers=127.20.207.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:16:42.991981 22755 heartbeater.cc:347] Connected to a master server at 127.20.207.62:41817
I20240502 08:16:42.992213 22755 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:42.993031 22755 heartbeater.cc:511] Master 127.20.207.62:41817 requested a full tablet report, sending...
I20240502 08:16:42.995210 22567 ts_manager.cc:194] Registered new tserver with Master: 00844c1f5dba4c0bbe65483833cadf96 (127.20.207.1:36851)
I20240502 08:16:42.996667 22567 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.1:49369
W20240502 08:16:43.289119 22759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:43.289372 22759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:43.289453 22759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:43.296504 22759 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:43.296605 22759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:43.296770 22759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.2
I20240502 08:16:43.305119 22759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.207.2
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41817
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.2
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:43.306340 22759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:43.307910 22759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:43.318735 22764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:43.319193 22767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:43.319841 22765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:43.320514 22759 server_base.cc:1034] running on GCE node
I20240502 08:16:43.321056 22759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:43.321923 22759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:43.323110 22759 hybrid_clock.cc:648] HybridClock initialized: now 1714637803323066 us; error 41 us; skew 500 ppm
I20240502 08:16:43.323413 22759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:43.325358 22759 webserver.cc:460] Webserver started at http://127.20.207.2:39321/ using document root <none> and password file <none>
I20240502 08:16:43.326031 22759 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:43.326164 22759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:43.326493 22759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:43.328832 22759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7dd335a774e54f849d8d4e19546b9fe9"
format_stamp: "Formatted at 2024-05-02 08:16:43 on dist-test-slave-169q"
I20240502 08:16:43.329563 22759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7dd335a774e54f849d8d4e19546b9fe9"
format_stamp: "Formatted at 2024-05-02 08:16:43 on dist-test-slave-169q"
I20240502 08:16:43.334465 22759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 08:16:43.337397 22773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:43.338618 22759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:16:43.338750 22759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7dd335a774e54f849d8d4e19546b9fe9"
format_stamp: "Formatted at 2024-05-02 08:16:43 on dist-test-slave-169q"
I20240502 08:16:43.338917 22759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:43.351227 22759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:43.351920 22759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:43.352172 22759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:43.352938 22759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:43.354210 22759 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:43.354301 22759 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:43.354403 22759 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:43.354473 22759 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:43.383708 22759 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.2:41893
I20240502 08:16:43.383764 22886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.2:41893 every 8 connection(s)
I20240502 08:16:43.384920 22759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:16:43.392973 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 22759
I20240502 08:16:43.393133 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:16:43.396270 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.3:0
--local_ip_for_outbound_sockets=127.20.207.3
--webserver_interface=127.20.207.3
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41817
--builtin_ntp_servers=127.20.207.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:16:43.396569 22887 heartbeater.cc:347] Connected to a master server at 127.20.207.62:41817
I20240502 08:16:43.396828 22887 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:43.397605 22887 heartbeater.cc:511] Master 127.20.207.62:41817 requested a full tablet report, sending...
I20240502 08:16:43.399077 22567 ts_manager.cc:194] Registered new tserver with Master: 7dd335a774e54f849d8d4e19546b9fe9 (127.20.207.2:41893)
I20240502 08:16:43.400144 22567 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.2:49967
W20240502 08:16:43.703244 22890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:43.703534 22890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:43.703655 22890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:43.710568 22890 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:43.710701 22890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:43.710853 22890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.3
I20240502 08:16:43.719805 22890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.207.3
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41817
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.3
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:43.721062 22890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:43.722744 22890 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:43.733840 22895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:43.734155 22898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:43.734745 22896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:43.735348 22890 server_base.cc:1034] running on GCE node
I20240502 08:16:43.735821 22890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:43.736845 22890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:43.738119 22890 hybrid_clock.cc:648] HybridClock initialized: now 1714637803738054 us; error 52 us; skew 500 ppm
I20240502 08:16:43.738437 22890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:43.740525 22890 webserver.cc:460] Webserver started at http://127.20.207.3:33503/ using document root <none> and password file <none>
I20240502 08:16:43.741200 22890 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:43.741305 22890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:43.741626 22890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:43.744107 22890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9676df6247d74139b91232fee08f3b5e"
format_stamp: "Formatted at 2024-05-02 08:16:43 on dist-test-slave-169q"
I20240502 08:16:43.744829 22890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9676df6247d74139b91232fee08f3b5e"
format_stamp: "Formatted at 2024-05-02 08:16:43 on dist-test-slave-169q"
I20240502 08:16:43.749401 22890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 08:16:43.752458 22904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:43.753679 22890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:43.753814 22890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9676df6247d74139b91232fee08f3b5e"
format_stamp: "Formatted at 2024-05-02 08:16:43 on dist-test-slave-169q"
I20240502 08:16:43.753988 22890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:43.769070 22890 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:43.769804 22890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:43.770087 22890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:43.771420 22890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:43.772763 22890 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:43.772842 22890 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:43.772953 22890 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:43.773022 22890 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:43.803429 22890 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.3:45179
I20240502 08:16:43.803474 23017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.3:45179 every 8 connection(s)
I20240502 08:16:43.804651 22890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:16:43.814144 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 22890
I20240502 08:16:43.814301 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:16:43.816347 23018 heartbeater.cc:347] Connected to a master server at 127.20.207.62:41817
I20240502 08:16:43.816666 23018 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:43.817497 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.4:0
--local_ip_for_outbound_sockets=127.20.207.4
--webserver_interface=127.20.207.4
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41817
--builtin_ntp_servers=127.20.207.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:16:43.817631 23018 heartbeater.cc:511] Master 127.20.207.62:41817 requested a full tablet report, sending...
I20240502 08:16:43.819116 22567 ts_manager.cc:194] Registered new tserver with Master: 9676df6247d74139b91232fee08f3b5e (127.20.207.3:45179)
I20240502 08:16:43.819864 22567 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.3:55723
I20240502 08:16:43.999296 22755 heartbeater.cc:503] Master 127.20.207.62:41817 was elected leader, sending a full tablet report...
W20240502 08:16:44.135677 23021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:44.136006 23021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:44.136143 23021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:44.143688 23021 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:44.143842 23021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:44.144016 23021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.4
I20240502 08:16:44.152809 23021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.207.4
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41817
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.4
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:44.154242 23021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:44.155951 23021 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:44.167833 23026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:44.168331 23027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:44.168763 23029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:44.169553 23021 server_base.cc:1034] running on GCE node
I20240502 08:16:44.170204 23021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:44.171136 23021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:44.172351 23021 hybrid_clock.cc:648] HybridClock initialized: now 1714637804172305 us; error 42 us; skew 500 ppm
I20240502 08:16:44.172660 23021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:44.174842 23021 webserver.cc:460] Webserver started at http://127.20.207.4:38129/ using document root <none> and password file <none>
I20240502 08:16:44.175590 23021 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:44.175705 23021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:44.176023 23021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:44.178617 23021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "f70d9450bce44e59b4612cbafab0f820"
format_stamp: "Formatted at 2024-05-02 08:16:44 on dist-test-slave-169q"
I20240502 08:16:44.179378 23021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "f70d9450bce44e59b4612cbafab0f820"
format_stamp: "Formatted at 2024-05-02 08:16:44 on dist-test-slave-169q"
I20240502 08:16:44.184625 23021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:16:44.188010 23035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:44.189265 23021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:44.189414 23021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f70d9450bce44e59b4612cbafab0f820"
format_stamp: "Formatted at 2024-05-02 08:16:44 on dist-test-slave-169q"
I20240502 08:16:44.189599 23021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:44.205742 23021 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:44.206624 23021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:44.206879 23021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:44.207764 23021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:44.209240 23021 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:44.209331 23021 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:44.209414 23021 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:44.209468 23021 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:44.239694 23021 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.4:40633
I20240502 08:16:44.239739 23148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.4:40633 every 8 connection(s)
I20240502 08:16:44.241003 23021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:16:44.245167 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 23021
I20240502 08:16:44.245350 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:16:44.248366 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.5:0
--local_ip_for_outbound_sockets=127.20.207.5
--webserver_interface=127.20.207.5
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41817
--builtin_ntp_servers=127.20.207.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:16:44.253365 23149 heartbeater.cc:347] Connected to a master server at 127.20.207.62:41817
I20240502 08:16:44.253644 23149 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:44.254442 23149 heartbeater.cc:511] Master 127.20.207.62:41817 requested a full tablet report, sending...
I20240502 08:16:44.255787 22567 ts_manager.cc:194] Registered new tserver with Master: f70d9450bce44e59b4612cbafab0f820 (127.20.207.4:40633)
I20240502 08:16:44.256940 22567 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.4:44345
I20240502 08:16:44.402834 22887 heartbeater.cc:503] Master 127.20.207.62:41817 was elected leader, sending a full tablet report...
W20240502 08:16:44.580096 23152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:44.580401 23152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:44.580528 23152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:44.588415 23152 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:44.588541 23152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:44.588697 23152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.5
I20240502 08:16:44.598076 23152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.207.5
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41817
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.5
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:44.599455 23152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:44.601176 23152 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:44.613571 23157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:44.614264 23160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:44.614950 23158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:44.620537 23152 server_base.cc:1034] running on GCE node
I20240502 08:16:44.621264 23152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:44.622349 23152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:44.623562 23152 hybrid_clock.cc:648] HybridClock initialized: now 1714637804623518 us; error 45 us; skew 500 ppm
I20240502 08:16:44.623876 23152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:44.626178 23152 webserver.cc:460] Webserver started at http://127.20.207.5:45481/ using document root <none> and password file <none>
I20240502 08:16:44.626936 23152 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:44.627053 23152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:44.627394 23152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:44.629992 23152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e530575b769046d3ba75edeea0d920ad"
format_stamp: "Formatted at 2024-05-02 08:16:44 on dist-test-slave-169q"
I20240502 08:16:44.630826 23152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e530575b769046d3ba75edeea0d920ad"
format_stamp: "Formatted at 2024-05-02 08:16:44 on dist-test-slave-169q"
I20240502 08:16:44.636138 23152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:16:44.639683 23166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:44.641060 23152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 08:16:44.641219 23152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e530575b769046d3ba75edeea0d920ad"
format_stamp: "Formatted at 2024-05-02 08:16:44 on dist-test-slave-169q"
I20240502 08:16:44.641429 23152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:44.672747 23152 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:44.673678 23152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:44.673966 23152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:44.675040 23152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:44.676540 23152 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:44.676628 23152 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:44.676710 23152 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:44.676761 23152 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:44.709606 23152 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.5:33069
I20240502 08:16:44.709662 23278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.5:33069 every 8 connection(s)
I20240502 08:16:44.711046 23152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:16:44.717569 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 23152
I20240502 08:16:44.717756 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:16:44.724716 23279 heartbeater.cc:347] Connected to a master server at 127.20.207.62:41817
I20240502 08:16:44.725004 23279 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:44.725816 23279 heartbeater.cc:511] Master 127.20.207.62:41817 requested a full tablet report, sending...
I20240502 08:16:44.727213 22567 ts_manager.cc:194] Registered new tserver with Master: e530575b769046d3ba75edeea0d920ad (127.20.207.5:33069)
I20240502 08:16:44.728003 22567 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.5:50363
I20240502 08:16:44.734130 21308 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 08:16:44.750193 22567 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48044:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 08:16:44.782109 22822 tablet_service.cc:1467] Processing CreateTablet for tablet 74e0c63e9b4c4dc58c2a86ec73a235e3 (DEFAULT_TABLE table=TestTable [id=1e6cfe52b60e46bb80bf16b69ec10a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.782130 22953 tablet_service.cc:1467] Processing CreateTablet for tablet 74e0c63e9b4c4dc58c2a86ec73a235e3 (DEFAULT_TABLE table=TestTable [id=1e6cfe52b60e46bb80bf16b69ec10a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.782130 23214 tablet_service.cc:1467] Processing CreateTablet for tablet 74e0c63e9b4c4dc58c2a86ec73a235e3 (DEFAULT_TABLE table=TestTable [id=1e6cfe52b60e46bb80bf16b69ec10a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.783428 22822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74e0c63e9b4c4dc58c2a86ec73a235e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.783515 22953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74e0c63e9b4c4dc58c2a86ec73a235e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.783514 23214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74e0c63e9b4c4dc58c2a86ec73a235e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.798429 23297 tablet_bootstrap.cc:492] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9: Bootstrap starting.
I20240502 08:16:44.798911 23298 tablet_bootstrap.cc:492] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 9676df6247d74139b91232fee08f3b5e: Bootstrap starting.
I20240502 08:16:44.800521 23299 tablet_bootstrap.cc:492] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad: Bootstrap starting.
I20240502 08:16:44.803056 23297 tablet_bootstrap.cc:654] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.803822 23298 tablet_bootstrap.cc:654] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 9676df6247d74139b91232fee08f3b5e: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.805081 23298 log.cc:824] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 9676df6247d74139b91232fee08f3b5e: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:44.806144 23297 log.cc:824] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:44.806891 23299 tablet_bootstrap.cc:654] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.808137 23299 log.cc:824] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:44.812222 23299 tablet_bootstrap.cc:492] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad: No bootstrap required, opened a new log
I20240502 08:16:44.812548 23299 ts_tablet_manager.cc:1367] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20240502 08:16:44.815672 23298 tablet_bootstrap.cc:492] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 9676df6247d74139b91232fee08f3b5e: No bootstrap required, opened a new log
I20240502 08:16:44.816037 23298 ts_tablet_manager.cc:1367] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 9676df6247d74139b91232fee08f3b5e: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.010s
I20240502 08:16:44.820212 23299 raft_consensus.cc:348] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9676df6247d74139b91232fee08f3b5e" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45179 } } peers { permanent_uuid: "7dd335a774e54f849d8d4e19546b9fe9" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 41893 } } peers { permanent_uuid: "e530575b769046d3ba75edeea0d920ad" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 33069 } }
I20240502 08:16:44.820590 23299 raft_consensus.cc:374] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.820688 23299 raft_consensus.cc:729] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e530575b769046d3ba75edeea0d920ad, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.820878 23297 tablet_bootstrap.cc:492] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9: No bootstrap required, opened a new log
I20240502 08:16:44.821273 23297 ts_tablet_manager.cc:1367] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.009s
I20240502 08:16:44.821461 23299 consensus_queue.cc:260] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9676df6247d74139b91232fee08f3b5e" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45179 } } peers { permanent_uuid: "7dd335a774e54f849d8d4e19546b9fe9" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 41893 } } peers { permanent_uuid: "e530575b769046d3ba75edeea0d920ad" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 33069 } }
I20240502 08:16:44.822520 23018 heartbeater.cc:503] Master 127.20.207.62:41817 was elected leader, sending a full tablet report...
I20240502 08:16:44.823196 23299 ts_tablet_manager.cc:1398] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20240502 08:16:44.823676 23279 heartbeater.cc:503] Master 127.20.207.62:41817 was elected leader, sending a full tablet report...
I20240502 08:16:44.828262 23298 raft_consensus.cc:348] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 9676df6247d74139b91232fee08f3b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9676df6247d74139b91232fee08f3b5e" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45179 } } peers { permanent_uuid: "7dd335a774e54f849d8d4e19546b9fe9" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 41893 } } peers { permanent_uuid: "e530575b769046d3ba75edeea0d920ad" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 33069 } }
I20240502 08:16:44.828716 23298 raft_consensus.cc:374] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 9676df6247d74139b91232fee08f3b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.828845 23298 raft_consensus.cc:729] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 9676df6247d74139b91232fee08f3b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9676df6247d74139b91232fee08f3b5e, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.829532 23298 consensus_queue.cc:260] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 9676df6247d74139b91232fee08f3b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9676df6247d74139b91232fee08f3b5e" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45179 } } peers { permanent_uuid: "7dd335a774e54f849d8d4e19546b9fe9" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 41893 } } peers { permanent_uuid: "e530575b769046d3ba75edeea0d920ad" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 33069 } }
I20240502 08:16:44.830243 23297 raft_consensus.cc:348] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9676df6247d74139b91232fee08f3b5e" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45179 } } peers { permanent_uuid: "7dd335a774e54f849d8d4e19546b9fe9" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 41893 } } peers { permanent_uuid: "e530575b769046d3ba75edeea0d920ad" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 33069 } }
I20240502 08:16:44.830710 23297 raft_consensus.cc:374] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.830816 23297 raft_consensus.cc:729] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dd335a774e54f849d8d4e19546b9fe9, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.831383 23298 ts_tablet_manager.cc:1398] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 9676df6247d74139b91232fee08f3b5e: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20240502 08:16:44.831590 23297 consensus_queue.cc:260] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9676df6247d74139b91232fee08f3b5e" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45179 } } peers { permanent_uuid: "7dd335a774e54f849d8d4e19546b9fe9" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 41893 } } peers { permanent_uuid: "e530575b769046d3ba75edeea0d920ad" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 33069 } }
I20240502 08:16:44.833204 23297 ts_tablet_manager.cc:1398] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20240502 08:16:44.887441 22888 tablet.cc:2369] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:16:44.962832 23280 tablet.cc:2369] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:16:45.057019 23019 tablet.cc:2369] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 9676df6247d74139b91232fee08f3b5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:45.180881 23303 raft_consensus.cc:482] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:45.181183 23303 raft_consensus.cc:504] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9676df6247d74139b91232fee08f3b5e" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45179 } } peers { permanent_uuid: "7dd335a774e54f849d8d4e19546b9fe9" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 41893 } } peers { permanent_uuid: "e530575b769046d3ba75edeea0d920ad" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 33069 } }
I20240502 08:16:45.182866 23303 leader_election.cc:288] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9676df6247d74139b91232fee08f3b5e (127.20.207.3:45179), 7dd335a774e54f849d8d4e19546b9fe9 (127.20.207.2:41893)
I20240502 08:16:45.189534 22973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e0c63e9b4c4dc58c2a86ec73a235e3" candidate_uuid: "e530575b769046d3ba75edeea0d920ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9676df6247d74139b91232fee08f3b5e" is_pre_election: true
I20240502 08:16:45.190081 23305 raft_consensus.cc:482] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:45.190135 22973 raft_consensus.cc:2454] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 9676df6247d74139b91232fee08f3b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e530575b769046d3ba75edeea0d920ad in term 0.
I20240502 08:16:45.190127 22842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e0c63e9b4c4dc58c2a86ec73a235e3" candidate_uuid: "e530575b769046d3ba75edeea0d920ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dd335a774e54f849d8d4e19546b9fe9" is_pre_election: true
I20240502 08:16:45.190351 23305 raft_consensus.cc:504] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9676df6247d74139b91232fee08f3b5e" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45179 } } peers { permanent_uuid: "7dd335a774e54f849d8d4e19546b9fe9" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 41893 } } peers { permanent_uuid: "e530575b769046d3ba75edeea0d920ad" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 33069 } }
I20240502 08:16:45.190806 23167 leader_election.cc:302] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9676df6247d74139b91232fee08f3b5e, e530575b769046d3ba75edeea0d920ad; no voters:
I20240502 08:16:45.191043 22842 raft_consensus.cc:2454] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e530575b769046d3ba75edeea0d920ad in term 0.
I20240502 08:16:45.191201 23303 raft_consensus.cc:2781] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:45.191407 23303 raft_consensus.cc:482] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:45.191563 23303 raft_consensus.cc:3037] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:45.192421 23305 leader_election.cc:288] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9676df6247d74139b91232fee08f3b5e (127.20.207.3:45179), e530575b769046d3ba75edeea0d920ad (127.20.207.5:33069)
I20240502 08:16:45.194622 23303 raft_consensus.cc:504] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9676df6247d74139b91232fee08f3b5e" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45179 } } peers { permanent_uuid: "7dd335a774e54f849d8d4e19546b9fe9" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 41893 } } peers { permanent_uuid: "e530575b769046d3ba75edeea0d920ad" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 33069 } }
I20240502 08:16:45.195356 23303 leader_election.cc:288] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [CANDIDATE]: Term 1 election: Requested vote from peers 9676df6247d74139b91232fee08f3b5e (127.20.207.3:45179), 7dd335a774e54f849d8d4e19546b9fe9 (127.20.207.2:41893)
I20240502 08:16:45.195803 22973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e0c63e9b4c4dc58c2a86ec73a235e3" candidate_uuid: "e530575b769046d3ba75edeea0d920ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9676df6247d74139b91232fee08f3b5e"
I20240502 08:16:45.196014 22973 raft_consensus.cc:3037] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 9676df6247d74139b91232fee08f3b5e [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:45.196063 22842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e0c63e9b4c4dc58c2a86ec73a235e3" candidate_uuid: "e530575b769046d3ba75edeea0d920ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dd335a774e54f849d8d4e19546b9fe9"
I20240502 08:16:45.196265 22842 raft_consensus.cc:3037] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:45.199236 22973 raft_consensus.cc:2454] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 9676df6247d74139b91232fee08f3b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e530575b769046d3ba75edeea0d920ad in term 1.
I20240502 08:16:45.199672 23167 leader_election.cc:302] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9676df6247d74139b91232fee08f3b5e, e530575b769046d3ba75edeea0d920ad; no voters:
I20240502 08:16:45.200054 22842 raft_consensus.cc:2454] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e530575b769046d3ba75edeea0d920ad in term 1.
I20240502 08:16:45.200565 23303 raft_consensus.cc:2781] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:45.201146 23303 raft_consensus.cc:686] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [term 1 LEADER]: Becoming Leader. State: Replica: e530575b769046d3ba75edeea0d920ad, State: Running, Role: LEADER
I20240502 08:16:45.201218 22973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e0c63e9b4c4dc58c2a86ec73a235e3" candidate_uuid: "7dd335a774e54f849d8d4e19546b9fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9676df6247d74139b91232fee08f3b5e" is_pre_election: true
I20240502 08:16:45.201508 22973 raft_consensus.cc:2379] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 9676df6247d74139b91232fee08f3b5e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7dd335a774e54f849d8d4e19546b9fe9 in current term 1: Already voted for candidate e530575b769046d3ba75edeea0d920ad in this term.
I20240502 08:16:45.202033 23234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e0c63e9b4c4dc58c2a86ec73a235e3" candidate_uuid: "7dd335a774e54f849d8d4e19546b9fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e530575b769046d3ba75edeea0d920ad" is_pre_election: true
I20240502 08:16:45.202143 23303 consensus_queue.cc:237] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9676df6247d74139b91232fee08f3b5e" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45179 } } peers { permanent_uuid: "7dd335a774e54f849d8d4e19546b9fe9" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 41893 } } peers { permanent_uuid: "e530575b769046d3ba75edeea0d920ad" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 33069 } }
I20240502 08:16:45.202999 22775 leader_election.cc:302] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7dd335a774e54f849d8d4e19546b9fe9; no voters: 9676df6247d74139b91232fee08f3b5e, e530575b769046d3ba75edeea0d920ad
I20240502 08:16:45.203374 23305 raft_consensus.cc:2726] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P 7dd335a774e54f849d8d4e19546b9fe9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:45.206676 22565 catalog_manager.cc:5484] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad reported cstate change: term changed from 0 to 1, leader changed from <none> to e530575b769046d3ba75edeea0d920ad (127.20.207.5). New cstate: current_term: 1 leader_uuid: "e530575b769046d3ba75edeea0d920ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9676df6247d74139b91232fee08f3b5e" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7dd335a774e54f849d8d4e19546b9fe9" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 41893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e530575b769046d3ba75edeea0d920ad" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 33069 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:45.233109 21308 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 08:16:45.235141 21308 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 00844c1f5dba4c0bbe65483833cadf96 to finish bootstrapping
I20240502 08:16:45.244870 21308 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7dd335a774e54f849d8d4e19546b9fe9 to finish bootstrapping
I20240502 08:16:45.252030 21308 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9676df6247d74139b91232fee08f3b5e to finish bootstrapping
I20240502 08:16:45.259336 21308 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f70d9450bce44e59b4612cbafab0f820 to finish bootstrapping
I20240502 08:16:45.259851 23149 heartbeater.cc:503] Master 127.20.207.62:41817 was elected leader, sending a full tablet report...
I20240502 08:16:45.267688 21308 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e530575b769046d3ba75edeea0d920ad to finish bootstrapping
I20240502 08:16:45.274839 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 22627
I20240502 08:16:45.275803 22750 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:45.354154 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 22627
I20240502 08:16:45.632346 23303 consensus_queue.cc:1035] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9676df6247d74139b91232fee08f3b5e" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:45.640080 23303 consensus_queue.cc:1035] T 74e0c63e9b4c4dc58c2a86ec73a235e3 P e530575b769046d3ba75edeea0d920ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dd335a774e54f849d8d4e19546b9fe9" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 41893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:45.836735 22953 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:45.837258 22822 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:45.838078 23214 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:45.841984 23084 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:45.979089 22822 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:45.981505 22953 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:45.996104 23084 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:46.004184 23214 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:46.163466 22953 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:46.164779 22822 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:46.166060 23214 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:46.171170 23084 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:46.813318 22822 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:46.814522 23214 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:46.827142 23084 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:46.831575 22953 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:46.957293 22822 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:46.962363 23084 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:46.963233 23214 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:46.964108 22953 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:47.125089 22822 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:47.129118 22953 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:47.132882 23214 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:47.143225 23084 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:47.285199 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 22759
I20240502 08:16:47.286715 22881 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:47.385941 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 22759
I20240502 08:16:47.404332 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 22890
I20240502 08:16:47.405342 23012 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:47.503154 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 22890
I20240502 08:16:47.521219 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 23021
I20240502 08:16:47.522318 23144 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:47.616904 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 23021
I20240502 08:16:47.634410 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 23152
I20240502 08:16:47.635455 23273 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:47.733253 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 23152
I20240502 08:16:47.753397 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 22536
I20240502 08:16:47.754571 22598 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:47.870770 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 22536
2024-05-02T08:16:47Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (5923 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5924 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/1
I20240502 08:16:47.916026 21308 test_util.cc:268] Using random seed: 964006873
I20240502 08:16:47.919195 21308 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:16:47.919289 21308 ts_itest-base.cc:113] --------------
I20240502 08:16:47.919370 21308 ts_itest-base.cc:114] 7 tablet servers
I20240502 08:16:47.919423 21308 ts_itest-base.cc:115] 1 replicas per TS
I20240502 08:16:47.919479 21308 ts_itest-base.cc:116] --------------
2024-05-02T08:16:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:16:47Z Disabled control of system clock
I20240502 08:16:47.945017 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.207.62:36471
--webserver_interface=127.20.207.62
--webserver_port=0
--builtin_ntp_servers=127.20.207.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.207.62:36471
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 08:16:48.281173 23522 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 08:16:48.281577 23522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:48.281690 23522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:48.281807 23522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:48.289558 23522 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:48.289712 23522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:16:48.289820 23522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:48.289894 23522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:16:48.289986 23522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:16:48.299695 23522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.207.62:36471
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.207.62:36471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.207.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:48.301097 23522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:48.302945 23522 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:48.314103 23530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:48.314589 23527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:48.315384 23528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:48.316107 23522 server_base.cc:1034] running on GCE node
I20240502 08:16:48.316962 23522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:48.318393 23522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:48.319608 23522 hybrid_clock.cc:648] HybridClock initialized: now 1714637808319548 us; error 43 us; skew 500 ppm
I20240502 08:16:48.319926 23522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:48.327503 23522 webserver.cc:460] Webserver started at http://127.20.207.62:43565/ using document root <none> and password file <none>
I20240502 08:16:48.328325 23522 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:48.328464 23522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:48.328768 23522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:48.331562 23522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7eb46d820d5744549fe812ddd6fca0b0"
format_stamp: "Formatted at 2024-05-02 08:16:48 on dist-test-slave-169q"
I20240502 08:16:48.332306 23522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7eb46d820d5744549fe812ddd6fca0b0"
format_stamp: "Formatted at 2024-05-02 08:16:48 on dist-test-slave-169q"
I20240502 08:16:48.337710 23522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240502 08:16:48.341375 23536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:48.342833 23522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:48.343025 23522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7eb46d820d5744549fe812ddd6fca0b0"
format_stamp: "Formatted at 2024-05-02 08:16:48 on dist-test-slave-169q"
I20240502 08:16:48.343225 23522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:48.369691 23522 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:48.370630 23522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:48.370951 23522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:48.388343 23522 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.62:36471
I20240502 08:16:48.388374 23588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.62:36471 every 8 connection(s)
I20240502 08:16:48.389827 23522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:16:48.393692 23589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.394287 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 23522
I20240502 08:16:48.394507 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:16:48.405987 23589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0: Bootstrap starting.
I20240502 08:16:48.409474 23589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.410632 23589 log.cc:824] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:48.413085 23589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0: No bootstrap required, opened a new log
I20240502 08:16:48.419247 23589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb46d820d5744549fe812ddd6fca0b0" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 36471 } }
I20240502 08:16:48.419584 23589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.419683 23589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb46d820d5744549fe812ddd6fca0b0, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.420253 23589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb46d820d5744549fe812ddd6fca0b0" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 36471 } }
I20240502 08:16:48.420444 23589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:48.420583 23589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:48.420718 23589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.447206 23589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb46d820d5744549fe812ddd6fca0b0" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 36471 } }
I20240502 08:16:48.447892 23589 leader_election.cc:302] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eb46d820d5744549fe812ddd6fca0b0; no voters:
I20240502 08:16:48.448539 23589 leader_election.cc:288] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:48.448760 23594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:48.449472 23594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb46d820d5744549fe812ddd6fca0b0, State: Running, Role: LEADER
I20240502 08:16:48.450448 23594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb46d820d5744549fe812ddd6fca0b0" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 36471 } }
I20240502 08:16:48.451707 23589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:48.453496 23595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eb46d820d5744549fe812ddd6fca0b0. Latest consensus state: current_term: 1 leader_uuid: "7eb46d820d5744549fe812ddd6fca0b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb46d820d5744549fe812ddd6fca0b0" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 36471 } } }
I20240502 08:16:48.453537 23596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eb46d820d5744549fe812ddd6fca0b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb46d820d5744549fe812ddd6fca0b0" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 36471 } } }
I20240502 08:16:48.453986 23596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:48.453980 23595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:48.455951 23599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:16:48.461119 23599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:16:48.469724 23599 catalog_manager.cc:1347] Generated new cluster ID: 30a11c06ced44790b24e01a9ed40ba15
I20240502 08:16:48.469878 23599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:16:48.485076 23599 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:16:48.486644 23599 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:16:48.510201 23599 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7eb46d820d5744549fe812ddd6fca0b0: Generated new TSK 0
I20240502 08:16:48.511057 23599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:16:48.530920 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.1:0
--local_ip_for_outbound_sockets=127.20.207.1
--webserver_interface=127.20.207.1
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36471
--builtin_ntp_servers=127.20.207.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:16:48.863031 23614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:48.863320 23614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:48.863461 23614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:48.872550 23614 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:48.872680 23614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:48.872851 23614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.1
I20240502 08:16:48.882282 23614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.207.1
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.1
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:48.883545 23614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:48.885278 23614 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:48.900913 23619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:48.901573 23620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:48.902299 23622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:48.902946 23614 server_base.cc:1034] running on GCE node
I20240502 08:16:48.903518 23614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:48.904428 23614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:48.905637 23614 hybrid_clock.cc:648] HybridClock initialized: now 1714637808905576 us; error 56 us; skew 500 ppm
I20240502 08:16:48.905936 23614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:48.908311 23614 webserver.cc:460] Webserver started at http://127.20.207.1:46097/ using document root <none> and password file <none>
I20240502 08:16:48.909006 23614 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:48.909118 23614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:48.909422 23614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:48.911959 23614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3de510fd6a8a47a6acc24acba19b2364"
format_stamp: "Formatted at 2024-05-02 08:16:48 on dist-test-slave-169q"
I20240502 08:16:48.912771 23614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3de510fd6a8a47a6acc24acba19b2364"
format_stamp: "Formatted at 2024-05-02 08:16:48 on dist-test-slave-169q"
I20240502 08:16:48.918248 23614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 08:16:48.923103 23628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:48.924711 23614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:48.924909 23614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3de510fd6a8a47a6acc24acba19b2364"
format_stamp: "Formatted at 2024-05-02 08:16:48 on dist-test-slave-169q"
I20240502 08:16:48.925201 23614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:48.941015 23614 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:48.941776 23614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:48.942090 23614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:48.942998 23614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:48.944433 23614 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:48.944517 23614 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:48.944586 23614 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:48.944659 23614 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:48.978654 23614 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.1:41519
I20240502 08:16:48.978724 23740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.1:41519 every 8 connection(s)
I20240502 08:16:48.979990 23614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:16:48.983211 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 23614
I20240502 08:16:48.983464 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:16:48.986896 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.2:0
--local_ip_for_outbound_sockets=127.20.207.2
--webserver_interface=127.20.207.2
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36471
--builtin_ntp_servers=127.20.207.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:16:48.993106 23741 heartbeater.cc:347] Connected to a master server at 127.20.207.62:36471
I20240502 08:16:48.993374 23741 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:48.994174 23741 heartbeater.cc:511] Master 127.20.207.62:36471 requested a full tablet report, sending...
I20240502 08:16:48.996109 23553 ts_manager.cc:194] Registered new tserver with Master: 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519)
I20240502 08:16:48.997644 23553 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.1:58497
W20240502 08:16:49.301041 23745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:49.301318 23745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:49.301474 23745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:49.308650 23745 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:49.308749 23745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:49.308887 23745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.2
I20240502 08:16:49.317595 23745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.207.2
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.2
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:49.318986 23745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:49.320776 23745 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:49.332988 23750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:49.333590 23753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:49.334254 23751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:49.335062 23745 server_base.cc:1034] running on GCE node
I20240502 08:16:49.335600 23745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:49.336356 23745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:49.337529 23745 hybrid_clock.cc:648] HybridClock initialized: now 1714637809337474 us; error 51 us; skew 500 ppm
I20240502 08:16:49.337790 23745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:49.339880 23745 webserver.cc:460] Webserver started at http://127.20.207.2:37387/ using document root <none> and password file <none>
I20240502 08:16:49.340564 23745 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:49.340674 23745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:49.340960 23745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:49.343469 23745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "dd7173a12d4a43b7bc728c0d99db5384"
format_stamp: "Formatted at 2024-05-02 08:16:49 on dist-test-slave-169q"
I20240502 08:16:49.344201 23745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "dd7173a12d4a43b7bc728c0d99db5384"
format_stamp: "Formatted at 2024-05-02 08:16:49 on dist-test-slave-169q"
I20240502 08:16:49.349110 23745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20240502 08:16:49.352232 23759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:49.353467 23745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 08:16:49.353603 23745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "dd7173a12d4a43b7bc728c0d99db5384"
format_stamp: "Formatted at 2024-05-02 08:16:49 on dist-test-slave-169q"
I20240502 08:16:49.353767 23745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:49.382932 23745 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:49.383729 23745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:49.383982 23745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:49.384908 23745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:49.386375 23745 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:49.386484 23745 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:49.386600 23745 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:49.386672 23745 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:49.423283 23745 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.2:35263
I20240502 08:16:49.423343 23873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.2:35263 every 8 connection(s)
I20240502 08:16:49.424506 23745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:16:49.433099 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 23745
I20240502 08:16:49.433272 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:16:49.436427 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.3:0
--local_ip_for_outbound_sockets=127.20.207.3
--webserver_interface=127.20.207.3
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36471
--builtin_ntp_servers=127.20.207.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:16:49.437003 23874 heartbeater.cc:347] Connected to a master server at 127.20.207.62:36471
I20240502 08:16:49.437278 23874 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:49.438079 23874 heartbeater.cc:511] Master 127.20.207.62:36471 requested a full tablet report, sending...
I20240502 08:16:49.439435 23553 ts_manager.cc:194] Registered new tserver with Master: dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263)
I20240502 08:16:49.440212 23553 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.2:43331
W20240502 08:16:49.740734 23877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:49.740995 23877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:49.741083 23877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:49.747876 23877 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:49.747977 23877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:49.748091 23877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.3
I20240502 08:16:49.756624 23877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.207.3
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.3
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:49.757879 23877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:49.759608 23877 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:49.770915 23882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:49.771715 23883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:49.772285 23885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:49.773092 23877 server_base.cc:1034] running on GCE node
I20240502 08:16:49.773671 23877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:49.774592 23877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:49.775792 23877 hybrid_clock.cc:648] HybridClock initialized: now 1714637809775741 us; error 53 us; skew 500 ppm
I20240502 08:16:49.776090 23877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:49.778276 23877 webserver.cc:460] Webserver started at http://127.20.207.3:37435/ using document root <none> and password file <none>
I20240502 08:16:49.779034 23877 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:49.779150 23877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:49.779469 23877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:49.781919 23877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "39c4a2d0ccc94109b6bd2cac1792003f"
format_stamp: "Formatted at 2024-05-02 08:16:49 on dist-test-slave-169q"
I20240502 08:16:49.782701 23877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "39c4a2d0ccc94109b6bd2cac1792003f"
format_stamp: "Formatted at 2024-05-02 08:16:49 on dist-test-slave-169q"
I20240502 08:16:49.787413 23877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240502 08:16:49.790527 23891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:49.791719 23877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20240502 08:16:49.791862 23877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "39c4a2d0ccc94109b6bd2cac1792003f"
format_stamp: "Formatted at 2024-05-02 08:16:49 on dist-test-slave-169q"
I20240502 08:16:49.792035 23877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:49.808758 23877 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:49.809764 23877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:49.810110 23877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:49.811267 23877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:49.813161 23877 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:49.813247 23877 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:49.813361 23877 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:49.813432 23877 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:49.843991 23877 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.3:45649
I20240502 08:16:49.844040 24004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.3:45649 every 8 connection(s)
I20240502 08:16:49.845201 23877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:16:49.853399 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 23877
I20240502 08:16:49.853586 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:16:49.856992 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.4:0
--local_ip_for_outbound_sockets=127.20.207.4
--webserver_interface=127.20.207.4
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36471
--builtin_ntp_servers=127.20.207.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:16:49.857632 24005 heartbeater.cc:347] Connected to a master server at 127.20.207.62:36471
I20240502 08:16:49.857887 24005 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:49.858659 24005 heartbeater.cc:511] Master 127.20.207.62:36471 requested a full tablet report, sending...
I20240502 08:16:49.859944 23553 ts_manager.cc:194] Registered new tserver with Master: 39c4a2d0ccc94109b6bd2cac1792003f (127.20.207.3:45649)
I20240502 08:16:49.860682 23553 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.3:49479
I20240502 08:16:50.000264 23741 heartbeater.cc:503] Master 127.20.207.62:36471 was elected leader, sending a full tablet report...
W20240502 08:16:50.163709 24008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:50.163955 24008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:50.164032 24008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:50.170617 24008 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:50.170706 24008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:50.170806 24008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.4
I20240502 08:16:50.179068 24008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.207.4
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.4
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:50.180140 24008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:50.181716 24008 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:50.192363 24013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:50.192878 24014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:50.193425 24016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:50.194376 24008 server_base.cc:1034] running on GCE node
I20240502 08:16:50.194854 24008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:50.195617 24008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:50.196811 24008 hybrid_clock.cc:648] HybridClock initialized: now 1714637810196765 us; error 44 us; skew 500 ppm
I20240502 08:16:50.197073 24008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:50.199103 24008 webserver.cc:460] Webserver started at http://127.20.207.4:33821/ using document root <none> and password file <none>
I20240502 08:16:50.199756 24008 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:50.199862 24008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:50.200145 24008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:50.202644 24008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "67e133ffa16d4a7a9ce63f934ab783b0"
format_stamp: "Formatted at 2024-05-02 08:16:50 on dist-test-slave-169q"
I20240502 08:16:50.203348 24008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "67e133ffa16d4a7a9ce63f934ab783b0"
format_stamp: "Formatted at 2024-05-02 08:16:50 on dist-test-slave-169q"
I20240502 08:16:50.207847 24008 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240502 08:16:50.210763 24022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:50.211962 24008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:16:50.212103 24008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "67e133ffa16d4a7a9ce63f934ab783b0"
format_stamp: "Formatted at 2024-05-02 08:16:50 on dist-test-slave-169q"
I20240502 08:16:50.212284 24008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:50.228899 24008 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:50.229660 24008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:50.229960 24008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:50.230993 24008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:50.232430 24008 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:50.232522 24008 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:50.232650 24008 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:50.232720 24008 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:50.262776 24008 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.4:43335
I20240502 08:16:50.262828 24135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.4:43335 every 8 connection(s)
I20240502 08:16:50.263985 24008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:16:50.272714 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 24008
I20240502 08:16:50.272907 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:16:50.275521 24136 heartbeater.cc:347] Connected to a master server at 127.20.207.62:36471
I20240502 08:16:50.275830 24136 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:50.276214 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.5:0
--local_ip_for_outbound_sockets=127.20.207.5
--webserver_interface=127.20.207.5
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36471
--builtin_ntp_servers=127.20.207.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:16:50.276799 24136 heartbeater.cc:511] Master 127.20.207.62:36471 requested a full tablet report, sending...
I20240502 08:16:50.278316 23552 ts_manager.cc:194] Registered new tserver with Master: 67e133ffa16d4a7a9ce63f934ab783b0 (127.20.207.4:43335)
I20240502 08:16:50.279124 23552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.4:55283
I20240502 08:16:50.442749 23874 heartbeater.cc:503] Master 127.20.207.62:36471 was elected leader, sending a full tablet report...
W20240502 08:16:50.598281 24139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:50.598606 24139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:50.598743 24139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:50.606266 24139 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:50.606417 24139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:50.606582 24139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.5
I20240502 08:16:50.615746 24139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.207.5
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.5
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:50.617233 24139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:50.619092 24139 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:50.631951 24145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:50.632557 24144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:50.633172 24147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:50.634042 24139 server_base.cc:1034] running on GCE node
I20240502 08:16:50.634688 24139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:50.635618 24139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:50.636817 24139 hybrid_clock.cc:648] HybridClock initialized: now 1714637810636764 us; error 54 us; skew 500 ppm
I20240502 08:16:50.637131 24139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:50.639482 24139 webserver.cc:460] Webserver started at http://127.20.207.5:40265/ using document root <none> and password file <none>
I20240502 08:16:50.640250 24139 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:50.640370 24139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:50.640678 24139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:50.643411 24139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "962f20ef8dea4cac91c4bf7603a655c0"
format_stamp: "Formatted at 2024-05-02 08:16:50 on dist-test-slave-169q"
I20240502 08:16:50.644203 24139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "962f20ef8dea4cac91c4bf7603a655c0"
format_stamp: "Formatted at 2024-05-02 08:16:50 on dist-test-slave-169q"
I20240502 08:16:50.649708 24139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20240502 08:16:50.653471 24153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:50.654956 24139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240502 08:16:50.655119 24139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "962f20ef8dea4cac91c4bf7603a655c0"
format_stamp: "Formatted at 2024-05-02 08:16:50 on dist-test-slave-169q"
I20240502 08:16:50.655321 24139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:50.672734 24139 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:50.673611 24139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:50.673882 24139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:50.674890 24139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:50.676373 24139 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:50.676473 24139 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:50.676591 24139 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:50.676666 24139 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:50.707216 24139 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.5:44257
I20240502 08:16:50.707259 24266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.5:44257 every 8 connection(s)
I20240502 08:16:50.708693 24139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:16:50.714198 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 24139
I20240502 08:16:50.714419 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:16:50.718097 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.6:0
--local_ip_for_outbound_sockets=127.20.207.6
--webserver_interface=127.20.207.6
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36471
--builtin_ntp_servers=127.20.207.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:16:50.722875 24267 heartbeater.cc:347] Connected to a master server at 127.20.207.62:36471
I20240502 08:16:50.723156 24267 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:50.723966 24267 heartbeater.cc:511] Master 127.20.207.62:36471 requested a full tablet report, sending...
I20240502 08:16:50.725462 23552 ts_manager.cc:194] Registered new tserver with Master: 962f20ef8dea4cac91c4bf7603a655c0 (127.20.207.5:44257)
I20240502 08:16:50.726312 23552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.5:60181
I20240502 08:16:50.862895 24005 heartbeater.cc:503] Master 127.20.207.62:36471 was elected leader, sending a full tablet report...
W20240502 08:16:51.057372 24270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:51.057693 24270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:51.057840 24270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:51.065526 24270 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:51.065675 24270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:51.065835 24270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.6
I20240502 08:16:51.074976 24270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.207.6
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.6
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:51.076310 24270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:51.078099 24270 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:51.090519 24275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:51.091054 24278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:51.091899 24276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:51.092790 24270 server_base.cc:1034] running on GCE node
I20240502 08:16:51.093405 24270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:51.094391 24270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:51.095602 24270 hybrid_clock.cc:648] HybridClock initialized: now 1714637811095557 us; error 41 us; skew 500 ppm
I20240502 08:16:51.095976 24270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:51.098402 24270 webserver.cc:460] Webserver started at http://127.20.207.6:43415/ using document root <none> and password file <none>
I20240502 08:16:51.099241 24270 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:51.099375 24270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:51.099696 24270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:51.102448 24270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4"
format_stamp: "Formatted at 2024-05-02 08:16:51 on dist-test-slave-169q"
I20240502 08:16:51.103209 24270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4"
format_stamp: "Formatted at 2024-05-02 08:16:51 on dist-test-slave-169q"
I20240502 08:16:51.108486 24270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:16:51.111928 24284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:51.113273 24270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:51.113436 24270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4"
format_stamp: "Formatted at 2024-05-02 08:16:51 on dist-test-slave-169q"
I20240502 08:16:51.113641 24270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:51.129436 24270 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:51.130411 24270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:51.130740 24270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:51.131742 24270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:51.133249 24270 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:51.133344 24270 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:51.133451 24270 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:51.133517 24270 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:51.167243 24270 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.6:42387
I20240502 08:16:51.167291 24397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.6:42387 every 8 connection(s)
I20240502 08:16:51.168591 24270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:16:51.171161 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 24270
I20240502 08:16:51.171365 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 08:16:51.174890 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.7:0
--local_ip_for_outbound_sockets=127.20.207.7
--webserver_interface=127.20.207.7
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36471
--builtin_ntp_servers=127.20.207.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:16:51.182962 24398 heartbeater.cc:347] Connected to a master server at 127.20.207.62:36471
I20240502 08:16:51.183267 24398 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:51.184098 24398 heartbeater.cc:511] Master 127.20.207.62:36471 requested a full tablet report, sending...
I20240502 08:16:51.185643 23552 ts_manager.cc:194] Registered new tserver with Master: 4b439f9c3d53400e8e2dc7b5c0a724b4 (127.20.207.6:42387)
I20240502 08:16:51.186931 23552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.6:51745
I20240502 08:16:51.281625 24136 heartbeater.cc:503] Master 127.20.207.62:36471 was elected leader, sending a full tablet report...
W20240502 08:16:51.510079 24401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:51.510396 24401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:51.510525 24401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:51.518327 24401 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:51.518447 24401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:51.518606 24401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.7
I20240502 08:16:51.527740 24401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.207.7
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.7
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:51.528959 24401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:51.530782 24401 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:51.542265 24407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:51.542879 24406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:51.543678 24409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:51.544188 24401 server_base.cc:1034] running on GCE node
I20240502 08:16:51.544747 24401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:51.545644 24401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:51.546860 24401 hybrid_clock.cc:648] HybridClock initialized: now 1714637811546810 us; error 27 us; skew 500 ppm
I20240502 08:16:51.547268 24401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:51.549428 24401 webserver.cc:460] Webserver started at http://127.20.207.7:32859/ using document root <none> and password file <none>
I20240502 08:16:51.550206 24401 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:51.550316 24401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:51.550629 24401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:51.553099 24401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "cfb4a4db281241f5940203fe907dea50"
format_stamp: "Formatted at 2024-05-02 08:16:51 on dist-test-slave-169q"
I20240502 08:16:51.553922 24401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "cfb4a4db281241f5940203fe907dea50"
format_stamp: "Formatted at 2024-05-02 08:16:51 on dist-test-slave-169q"
I20240502 08:16:51.559144 24401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 08:16:51.562363 24415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:51.563683 24401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 08:16:51.563827 24401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "cfb4a4db281241f5940203fe907dea50"
format_stamp: "Formatted at 2024-05-02 08:16:51 on dist-test-slave-169q"
I20240502 08:16:51.564008 24401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:51.590389 24401 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:51.591214 24401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:51.591490 24401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:51.592417 24401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:51.593823 24401 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:51.593905 24401 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:51.594031 24401 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:51.594105 24401 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240502 08:16:51.624589 24401 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.7:39999
I20240502 08:16:51.624670 24527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.7:39999 every 8 connection(s)
I20240502 08:16:51.625865 24401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 08:16:51.635143 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 24401
I20240502 08:16:51.635380 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 08:16:51.638206 24528 heartbeater.cc:347] Connected to a master server at 127.20.207.62:36471
I20240502 08:16:51.638530 24528 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:51.639402 24528 heartbeater.cc:511] Master 127.20.207.62:36471 requested a full tablet report, sending...
I20240502 08:16:51.640632 23552 ts_manager.cc:194] Registered new tserver with Master: cfb4a4db281241f5940203fe907dea50 (127.20.207.7:39999)
I20240502 08:16:51.641356 23552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.7:35155
I20240502 08:16:51.651494 21308 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 08:16:51.666914 23552 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48592:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 08:16:51.693265 23676 tablet_service.cc:1467] Processing CreateTablet for tablet eff43ec5b0ea4198ae23d57f2c4cb2c9 (DEFAULT_TABLE table=TestTable [id=3b4c053dcc94451bae4c42fa985e8559]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.694514 23676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff43ec5b0ea4198ae23d57f2c4cb2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.704772 24542 tablet_bootstrap.cc:492] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364: Bootstrap starting.
I20240502 08:16:51.708312 24542 tablet_bootstrap.cc:654] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:51.709657 24542 log.cc:824] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:51.712239 24542 tablet_bootstrap.cc:492] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364: No bootstrap required, opened a new log
I20240502 08:16:51.712533 24542 ts_tablet_manager.cc:1367] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 08:16:51.719267 24542 raft_consensus.cc:348] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:51.719594 24542 raft_consensus.cc:374] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:51.719700 24542 raft_consensus.cc:729] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Initialized, Role: FOLLOWER
I20240502 08:16:51.720304 24542 consensus_queue.cc:260] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:51.720485 24542 raft_consensus.cc:388] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:51.720616 24542 raft_consensus.cc:482] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:51.720753 24542 raft_consensus.cc:3037] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:51.723169 24542 raft_consensus.cc:504] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:51.723664 24542 leader_election.cc:302] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3de510fd6a8a47a6acc24acba19b2364; no voters:
I20240502 08:16:51.724241 24542 leader_election.cc:288] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:51.724396 24544 raft_consensus.cc:2781] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:51.725020 24544 raft_consensus.cc:686] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Becoming Leader. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Running, Role: LEADER
I20240502 08:16:51.725565 24544 consensus_queue.cc:237] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:51.726019 24542 ts_tablet_manager.cc:1398] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20240502 08:16:51.729423 24267 heartbeater.cc:503] Master 127.20.207.62:36471 was elected leader, sending a full tablet report...
I20240502 08:16:51.730923 23552 catalog_manager.cc:5484] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:51.750782 21308 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 08:16:51.752643 21308 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3de510fd6a8a47a6acc24acba19b2364 to finish bootstrapping
I20240502 08:16:51.761261 21308 ts_itest-base.cc:233] Waiting for 0 tablets on tserver dd7173a12d4a43b7bc728c0d99db5384 to finish bootstrapping
I20240502 08:16:51.770061 21308 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 39c4a2d0ccc94109b6bd2cac1792003f to finish bootstrapping
I20240502 08:16:51.778685 21308 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 67e133ffa16d4a7a9ce63f934ab783b0 to finish bootstrapping
I20240502 08:16:51.787329 21308 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 962f20ef8dea4cac91c4bf7603a655c0 to finish bootstrapping
I20240502 08:16:51.795289 21308 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4b439f9c3d53400e8e2dc7b5c0a724b4 to finish bootstrapping
I20240502 08:16:51.803011 21308 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cfb4a4db281241f5940203fe907dea50 to finish bootstrapping
I20240502 08:16:51.811425 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 23877
I20240502 08:16:51.812363 24000 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:51.890260 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 23877
W20240502 08:16:51.906526 23537 connection.cc:535] server connection from 127.20.207.3:49479 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:16:51.906811 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 24008
I20240502 08:16:51.907997 24131 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:51.989753 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 24008
I20240502 08:16:52.005319 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 24139
I20240502 08:16:52.006446 24262 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:52.084652 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 24139
I20240502 08:16:52.100039 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 24270
I20240502 08:16:52.101202 24393 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:52.181421 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 24270
I20240502 08:16:52.196799 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 24401
I20240502 08:16:52.197876 24523 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:52.277158 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 24401
I20240502 08:16:56.044819 23552 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48592:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:16:56.055838 23676 tablet_service.cc:1467] Processing CreateTablet for tablet f6dd3894b6aa489f92fcb1aa383b2a58 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c281f5d4d5124b60befcabf1e97b35a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.056275 23676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6dd3894b6aa489f92fcb1aa383b2a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.061389 24567 tablet_bootstrap.cc:492] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364: Bootstrap starting.
I20240502 08:16:56.062799 24567 tablet_bootstrap.cc:654] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.064620 24567 tablet_bootstrap.cc:492] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364: No bootstrap required, opened a new log
I20240502 08:16:56.064754 24567 ts_tablet_manager.cc:1367] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 08:16:56.064584 23807 tablet_service.cc:1467] Processing CreateTablet for tablet f214664b085a4ac0a10484acbd525142 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c281f5d4d5124b60befcabf1e97b35a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.064631 23808 tablet_service.cc:1467] Processing CreateTablet for tablet bf84e851bd9e41acb2fc657fb516cb6e (DEFAULT_TABLE table=rebalance_test_table_0 [id=c281f5d4d5124b60befcabf1e97b35a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.065522 24567 raft_consensus.cc:348] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:56.065663 24567 raft_consensus.cc:374] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.065754 24567 raft_consensus.cc:729] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.065969 24567 consensus_queue.cc:260] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:56.066162 24567 raft_consensus.cc:388] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:56.066242 24567 raft_consensus.cc:482] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:56.066349 24567 raft_consensus.cc:3037] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.066360 23807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f214664b085a4ac0a10484acbd525142. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.066795 23808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf84e851bd9e41acb2fc657fb516cb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.068085 24567 raft_consensus.cc:504] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:56.068310 24567 leader_election.cc:302] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3de510fd6a8a47a6acc24acba19b2364; no voters:
I20240502 08:16:56.068743 24567 leader_election.cc:288] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:56.068892 24569 raft_consensus.cc:2781] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:56.069618 24567 ts_tablet_manager.cc:1398] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20240502 08:16:56.071722 24569 raft_consensus.cc:686] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Becoming Leader. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Running, Role: LEADER
I20240502 08:16:56.072118 24569 consensus_queue.cc:237] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:56.079212 23552 catalog_manager.cc:5484] T f6dd3894b6aa489f92fcb1aa383b2a58 P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:56.085399 24574 tablet_bootstrap.cc:492] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384: Bootstrap starting.
I20240502 08:16:56.088387 24574 tablet_bootstrap.cc:654] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.089381 24574 log.cc:824] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:56.092427 24574 tablet_bootstrap.cc:492] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384: No bootstrap required, opened a new log
I20240502 08:16:56.092741 24574 ts_tablet_manager.cc:1367] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 08:16:56.102372 24574 raft_consensus.cc:348] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:56.102919 24574 raft_consensus.cc:374] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.103049 24574 raft_consensus.cc:729] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.103860 24574 consensus_queue.cc:260] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:56.104131 24574 raft_consensus.cc:388] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:56.104285 24574 raft_consensus.cc:482] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:56.104465 24574 raft_consensus.cc:3037] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.107499 24574 raft_consensus.cc:504] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:56.108078 24574 leader_election.cc:302] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd7173a12d4a43b7bc728c0d99db5384; no voters:
I20240502 08:16:56.108733 24574 leader_election.cc:288] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:56.108865 24576 raft_consensus.cc:2781] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:56.109580 24576 raft_consensus.cc:686] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Becoming Leader. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Running, Role: LEADER
I20240502 08:16:56.110512 24576 consensus_queue.cc:237] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:56.110692 24574 ts_tablet_manager.cc:1398] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384: Time spent starting tablet: real 0.018s user 0.019s sys 0.000s
I20240502 08:16:56.111075 24574 tablet_bootstrap.cc:492] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384: Bootstrap starting.
I20240502 08:16:56.112780 24574 tablet_bootstrap.cc:654] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.114785 24574 tablet_bootstrap.cc:492] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384: No bootstrap required, opened a new log
I20240502 08:16:56.114961 24574 ts_tablet_manager.cc:1367] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:16:56.115787 24574 raft_consensus.cc:348] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:56.115898 24574 raft_consensus.cc:374] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.115953 24574 raft_consensus.cc:729] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.115801 23552 catalog_manager.cc:5484] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:56.116155 24574 consensus_queue.cc:260] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:56.116297 24574 raft_consensus.cc:388] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:56.116359 24574 raft_consensus.cc:482] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:56.116428 24574 raft_consensus.cc:3037] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.118697 24574 raft_consensus.cc:504] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:56.118994 24574 leader_election.cc:302] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd7173a12d4a43b7bc728c0d99db5384; no voters:
I20240502 08:16:56.119187 24574 leader_election.cc:288] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:56.119344 24576 raft_consensus.cc:2781] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:56.119619 24576 raft_consensus.cc:686] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Becoming Leader. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Running, Role: LEADER
I20240502 08:16:56.119669 24574 ts_tablet_manager.cc:1398] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 08:16:56.120010 24576 consensus_queue.cc:237] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:56.122859 23553 catalog_manager.cc:5484] T bf84e851bd9e41acb2fc657fb516cb6e P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } health_report { overall_health: HEALTHY } } }
W20240502 08:16:56.468783 24584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:56.469121 24584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:56.476154 24584 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 117.511 ms
time per row: 0.0587557 ms
I20240502 08:16:56.732267 23553 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48592:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:16:56.740753 23676 tablet_service.cc:1467] Processing CreateTablet for tablet db6049ba7a5640d4ac4dae656b2f3667 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6ca06ec7e9d49f88451cc03d5275d15]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.740981 23807 tablet_service.cc:1467] Processing CreateTablet for tablet 9a4dfe88384b446fa08524fa91e557b7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6ca06ec7e9d49f88451cc03d5275d15]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.741292 23676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db6049ba7a5640d4ac4dae656b2f3667. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.741436 23807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a4dfe88384b446fa08524fa91e557b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.741298 23675 tablet_service.cc:1467] Processing CreateTablet for tablet 7df5ab24ba3c42769dd4640220641996 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6ca06ec7e9d49f88451cc03d5275d15]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.741844 23675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df5ab24ba3c42769dd4640220641996. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.746389 24627 tablet_bootstrap.cc:492] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384: Bootstrap starting.
I20240502 08:16:56.747756 24627 tablet_bootstrap.cc:654] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.748526 24628 tablet_bootstrap.cc:492] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364: Bootstrap starting.
I20240502 08:16:56.749574 24627 tablet_bootstrap.cc:492] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384: No bootstrap required, opened a new log
I20240502 08:16:56.749712 24627 ts_tablet_manager.cc:1367] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:16:56.750181 24628 tablet_bootstrap.cc:654] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.750438 24627 raft_consensus.cc:348] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:56.750627 24627 raft_consensus.cc:374] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.750731 24627 raft_consensus.cc:729] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.750946 24627 consensus_queue.cc:260] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:56.751086 24627 raft_consensus.cc:388] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:56.751163 24627 raft_consensus.cc:482] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:56.751292 24627 raft_consensus.cc:3037] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.751931 24628 tablet_bootstrap.cc:492] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364: No bootstrap required, opened a new log
I20240502 08:16:56.752079 24628 ts_tablet_manager.cc:1367] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:16:56.752671 24628 raft_consensus.cc:348] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:56.752807 24628 raft_consensus.cc:374] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.752897 24628 raft_consensus.cc:729] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.753132 24628 consensus_queue.cc:260] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:56.753320 24628 raft_consensus.cc:388] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:56.753305 24627 raft_consensus.cc:504] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:56.753412 24628 raft_consensus.cc:482] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:56.753527 24628 raft_consensus.cc:3037] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.753546 24627 leader_election.cc:302] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd7173a12d4a43b7bc728c0d99db5384; no voters:
I20240502 08:16:56.753743 24627 leader_election.cc:288] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:56.753854 24576 raft_consensus.cc:2781] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:56.754307 24627 ts_tablet_manager.cc:1398] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20240502 08:16:56.754668 24576 raft_consensus.cc:686] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Becoming Leader. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Running, Role: LEADER
I20240502 08:16:56.754935 24576 consensus_queue.cc:237] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:56.756287 24628 raft_consensus.cc:504] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:56.756531 24628 leader_election.cc:302] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3de510fd6a8a47a6acc24acba19b2364; no voters:
I20240502 08:16:56.756824 24628 leader_election.cc:288] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:56.756942 24620 raft_consensus.cc:2781] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:56.757634 23553 catalog_manager.cc:5484] T 9a4dfe88384b446fa08524fa91e557b7 P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:56.757786 24620 raft_consensus.cc:686] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Becoming Leader. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Running, Role: LEADER
I20240502 08:16:56.757920 24628 ts_tablet_manager.cc:1398] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20240502 08:16:56.758163 24620 consensus_queue.cc:237] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:56.758461 24628 tablet_bootstrap.cc:492] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364: Bootstrap starting.
I20240502 08:16:56.760499 24628 tablet_bootstrap.cc:654] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.762135 23553 catalog_manager.cc:5484] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:56.767161 24628 tablet_bootstrap.cc:492] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364: No bootstrap required, opened a new log
I20240502 08:16:56.767292 24628 ts_tablet_manager.cc:1367] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20240502 08:16:56.767977 24628 raft_consensus.cc:348] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:56.768126 24628 raft_consensus.cc:374] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.768198 24628 raft_consensus.cc:729] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.768420 24628 consensus_queue.cc:260] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:56.768554 24628 raft_consensus.cc:388] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:56.768646 24628 raft_consensus.cc:482] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:56.768733 24628 raft_consensus.cc:3037] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.770576 24628 raft_consensus.cc:504] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:56.770779 24628 leader_election.cc:302] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3de510fd6a8a47a6acc24acba19b2364; no voters:
I20240502 08:16:56.770959 24628 leader_election.cc:288] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:56.771032 24620 raft_consensus.cc:2781] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:56.771247 24620 raft_consensus.cc:686] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Becoming Leader. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Running, Role: LEADER
I20240502 08:16:56.771416 24628 ts_tablet_manager.cc:1398] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:16:56.771459 24620 consensus_queue.cc:237] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:56.773365 23553 catalog_manager.cc:5484] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } health_report { overall_health: HEALTHY } } }
W20240502 08:16:57.099766 24638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:57.100076 24638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:57.107240 24638 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 116.641 ms
time per row: 0.0583205 ms
I20240502 08:16:57.365396 23551 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48592:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:16:57.374397 23808 tablet_service.cc:1467] Processing CreateTablet for tablet c9d6f55ce8f547e395d686838dbb7bd2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=caae7f99bbcf4a949362006ef7059b72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.374385 23807 tablet_service.cc:1467] Processing CreateTablet for tablet fb067e2ff17740958059cfef1e0d0f9d (DEFAULT_TABLE table=rebalance_test_table_2 [id=caae7f99bbcf4a949362006ef7059b72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.374873 23808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d6f55ce8f547e395d686838dbb7bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.374853 23675 tablet_service.cc:1467] Processing CreateTablet for tablet 9506dfea92e3488fbe504216a75c4798 (DEFAULT_TABLE table=rebalance_test_table_2 [id=caae7f99bbcf4a949362006ef7059b72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.375160 23807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb067e2ff17740958059cfef1e0d0f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.375236 23675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9506dfea92e3488fbe504216a75c4798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.380211 24680 tablet_bootstrap.cc:492] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364: Bootstrap starting.
I20240502 08:16:57.381760 24680 tablet_bootstrap.cc:654] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:57.382447 24681 tablet_bootstrap.cc:492] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384: Bootstrap starting.
I20240502 08:16:57.383895 24680 tablet_bootstrap.cc:492] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364: No bootstrap required, opened a new log
I20240502 08:16:57.384047 24680 ts_tablet_manager.cc:1367] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:16:57.384127 24681 tablet_bootstrap.cc:654] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:57.384903 24680 raft_consensus.cc:348] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:57.385128 24680 raft_consensus.cc:374] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:57.385223 24680 raft_consensus.cc:729] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Initialized, Role: FOLLOWER
I20240502 08:16:57.385458 24680 consensus_queue.cc:260] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:57.385659 24680 raft_consensus.cc:388] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:57.385756 24680 raft_consensus.cc:482] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:57.385864 24680 raft_consensus.cc:3037] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:57.386104 24681 tablet_bootstrap.cc:492] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384: No bootstrap required, opened a new log
I20240502 08:16:57.386238 24681 ts_tablet_manager.cc:1367] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:16:57.386989 24681 raft_consensus.cc:348] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:57.387157 24681 raft_consensus.cc:374] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:57.387248 24681 raft_consensus.cc:729] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Initialized, Role: FOLLOWER
I20240502 08:16:57.387473 24681 consensus_queue.cc:260] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:57.387681 24681 raft_consensus.cc:388] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:57.387763 24681 raft_consensus.cc:482] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:57.387863 24681 raft_consensus.cc:3037] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:57.388656 24680 raft_consensus.cc:504] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:57.388912 24680 leader_election.cc:302] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3de510fd6a8a47a6acc24acba19b2364; no voters:
I20240502 08:16:57.389112 24680 leader_election.cc:288] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:57.389235 24633 raft_consensus.cc:2781] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:57.389389 24633 raft_consensus.cc:686] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Becoming Leader. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Running, Role: LEADER
I20240502 08:16:57.389695 24633 consensus_queue.cc:237] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:57.389930 24680 ts_tablet_manager.cc:1398] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20240502 08:16:57.390592 24681 raft_consensus.cc:504] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:57.390853 24681 leader_election.cc:302] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd7173a12d4a43b7bc728c0d99db5384; no voters:
I20240502 08:16:57.391039 24681 leader_election.cc:288] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:57.391108 24576 raft_consensus.cc:2781] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:57.392570 24681 ts_tablet_manager.cc:1398] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 08:16:57.392827 24681 tablet_bootstrap.cc:492] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384: Bootstrap starting.
I20240502 08:16:57.393554 23551 catalog_manager.cc:5484] T 9506dfea92e3488fbe504216a75c4798 P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:57.394564 24681 tablet_bootstrap.cc:654] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:57.395457 24576 raft_consensus.cc:686] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Becoming Leader. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Running, Role: LEADER
I20240502 08:16:57.395793 24576 consensus_queue.cc:237] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:57.398718 24681 tablet_bootstrap.cc:492] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384: No bootstrap required, opened a new log
I20240502 08:16:57.398900 24681 ts_tablet_manager.cc:1367] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:16:57.399478 24681 raft_consensus.cc:348] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:57.399645 24681 raft_consensus.cc:374] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:57.400765 24681 raft_consensus.cc:729] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Initialized, Role: FOLLOWER
I20240502 08:16:57.401069 24681 consensus_queue.cc:260] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:57.401242 24681 raft_consensus.cc:388] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:57.401346 24681 raft_consensus.cc:482] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:57.401443 24681 raft_consensus.cc:3037] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:57.404014 24681 raft_consensus.cc:504] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:57.404237 24681 leader_election.cc:302] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd7173a12d4a43b7bc728c0d99db5384; no voters:
I20240502 08:16:57.404441 24681 leader_election.cc:288] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:57.404544 24685 raft_consensus.cc:2781] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:57.404697 24685 raft_consensus.cc:686] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Becoming Leader. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Running, Role: LEADER
I20240502 08:16:57.405009 24681 ts_tablet_manager.cc:1398] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 08:16:57.404953 24685 consensus_queue.cc:237] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:57.407490 23551 catalog_manager.cc:5484] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:57.412021 23551 catalog_manager.cc:5484] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } health_report { overall_health: HEALTHY } } }
W20240502 08:16:57.749085 24690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:57.749413 24690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:57.757221 24690 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 117.299 ms
time per row: 0.0586495 ms
I20240502 08:16:58.015663 23545 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48592:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:16:58.024585 23675 tablet_service.cc:1467] Processing CreateTablet for tablet 1c26c8082287402daf70ee9be5a4fb7f (DEFAULT_TABLE table=rebalance_test_table_3 [id=0a34792f1dc944fea6b6ce2a62a12693]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.024690 23807 tablet_service.cc:1467] Processing CreateTablet for tablet ac96d624768a4d26bc2cbc43d120d3d1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0a34792f1dc944fea6b6ce2a62a12693]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.024926 23676 tablet_service.cc:1467] Processing CreateTablet for tablet 8096d61335f8493594e55720d47489bd (DEFAULT_TABLE table=rebalance_test_table_3 [id=0a34792f1dc944fea6b6ce2a62a12693]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.025079 23675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c26c8082287402daf70ee9be5a4fb7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.025136 23807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac96d624768a4d26bc2cbc43d120d3d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.025271 23676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8096d61335f8493594e55720d47489bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.031785 24732 tablet_bootstrap.cc:492] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384: Bootstrap starting.
I20240502 08:16:58.033249 24732 tablet_bootstrap.cc:654] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.033947 24733 tablet_bootstrap.cc:492] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364: Bootstrap starting.
I20240502 08:16:58.035576 24732 tablet_bootstrap.cc:492] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384: No bootstrap required, opened a new log
I20240502 08:16:58.035722 24732 ts_tablet_manager.cc:1367] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:16:58.035779 24733 tablet_bootstrap.cc:654] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.036325 24732 raft_consensus.cc:348] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:58.036476 24732 raft_consensus.cc:374] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.036576 24732 raft_consensus.cc:729] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.036790 24732 consensus_queue.cc:260] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:58.036948 24732 raft_consensus.cc:388] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:58.037022 24732 raft_consensus.cc:482] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:58.037125 24732 raft_consensus.cc:3037] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:58.037477 24733 tablet_bootstrap.cc:492] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364: No bootstrap required, opened a new log
I20240502 08:16:58.037604 24733 ts_tablet_manager.cc:1367] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 08:16:58.038396 24733 raft_consensus.cc:348] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:58.038537 24733 raft_consensus.cc:374] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.038610 24733 raft_consensus.cc:729] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.038843 24733 consensus_queue.cc:260] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:58.039083 24733 raft_consensus.cc:388] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:58.039166 24733 raft_consensus.cc:482] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:58.039290 24733 raft_consensus.cc:3037] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:58.039412 24732 raft_consensus.cc:504] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:58.039626 24732 leader_election.cc:302] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd7173a12d4a43b7bc728c0d99db5384; no voters:
I20240502 08:16:58.039827 24732 leader_election.cc:288] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:58.039930 24576 raft_consensus.cc:2781] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:58.040061 24576 raft_consensus.cc:686] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Becoming Leader. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Running, Role: LEADER
I20240502 08:16:58.040333 24732 ts_tablet_manager.cc:1398] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:16:58.040314 24576 consensus_queue.cc:237] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:58.041579 24733 raft_consensus.cc:504] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:58.041826 24733 leader_election.cc:302] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3de510fd6a8a47a6acc24acba19b2364; no voters:
I20240502 08:16:58.042079 24733 leader_election.cc:288] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:58.042148 24633 raft_consensus.cc:2781] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:58.042635 24633 raft_consensus.cc:686] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Becoming Leader. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Running, Role: LEADER
I20240502 08:16:58.042908 24633 consensus_queue.cc:237] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:58.043154 23545 catalog_manager.cc:5484] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:58.046037 23546 catalog_manager.cc:5484] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:58.046347 24733 ts_tablet_manager.cc:1398] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364: Time spent starting tablet: real 0.009s user 0.004s sys 0.005s
I20240502 08:16:58.046548 24733 tablet_bootstrap.cc:492] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364: Bootstrap starting.
I20240502 08:16:58.048108 24733 tablet_bootstrap.cc:654] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.049939 24733 tablet_bootstrap.cc:492] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364: No bootstrap required, opened a new log
I20240502 08:16:58.050103 24733 ts_tablet_manager.cc:1367] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 08:16:58.050662 24733 raft_consensus.cc:348] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:58.050808 24733 raft_consensus.cc:374] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.050879 24733 raft_consensus.cc:729] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.051076 24733 consensus_queue.cc:260] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:58.051211 24733 raft_consensus.cc:388] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:58.051321 24733 raft_consensus.cc:482] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:58.051409 24733 raft_consensus.cc:3037] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:58.053637 24733 raft_consensus.cc:504] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:58.053845 24733 leader_election.cc:302] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3de510fd6a8a47a6acc24acba19b2364; no voters:
I20240502 08:16:58.054054 24733 leader_election.cc:288] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:58.054132 24633 raft_consensus.cc:2781] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:58.054287 24633 raft_consensus.cc:686] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Becoming Leader. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Running, Role: LEADER
I20240502 08:16:58.054569 24733 ts_tablet_manager.cc:1398] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20240502 08:16:58.054582 24633 consensus_queue.cc:237] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:58.056742 23546 catalog_manager.cc:5484] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } health_report { overall_health: HEALTHY } } }
W20240502 08:16:58.397708 24742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:58.398108 24742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:58.405722 24742 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 138.619 ms
time per row: 0.0693094 ms
I20240502 08:16:58.697513 23545 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48592:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:16:58.706454 23807 tablet_service.cc:1467] Processing CreateTablet for tablet ce947f6507ea457da00437f718444e5b (DEFAULT_TABLE table=rebalance_test_table_4 [id=ca0b1080aa984ac38b4aa6099730ddce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.706606 23808 tablet_service.cc:1467] Processing CreateTablet for tablet 2c0157c3916c4012a6b8c4d38df7a342 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ca0b1080aa984ac38b4aa6099730ddce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.706635 23675 tablet_service.cc:1467] Processing CreateTablet for tablet b15e8c74b3b448538e315598c323601a (DEFAULT_TABLE table=rebalance_test_table_4 [id=ca0b1080aa984ac38b4aa6099730ddce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.706967 23807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce947f6507ea457da00437f718444e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.707022 23675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15e8c74b3b448538e315598c323601a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.707276 23808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c0157c3916c4012a6b8c4d38df7a342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.713683 24788 tablet_bootstrap.cc:492] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364: Bootstrap starting.
I20240502 08:16:58.714376 24789 tablet_bootstrap.cc:492] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384: Bootstrap starting.
I20240502 08:16:58.715859 24788 tablet_bootstrap.cc:654] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.716078 24789 tablet_bootstrap.cc:654] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.717886 24788 tablet_bootstrap.cc:492] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364: No bootstrap required, opened a new log
I20240502 08:16:58.718030 24789 tablet_bootstrap.cc:492] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384: No bootstrap required, opened a new log
I20240502 08:16:58.718052 24788 ts_tablet_manager.cc:1367] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 08:16:58.718173 24789 ts_tablet_manager.cc:1367] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240502 08:16:58.718686 24788 raft_consensus.cc:348] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:58.718837 24788 raft_consensus.cc:374] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.718775 24789 raft_consensus.cc:348] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:58.718943 24789 raft_consensus.cc:374] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.718961 24788 raft_consensus.cc:729] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.719046 24789 raft_consensus.cc:729] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.719208 24788 consensus_queue.cc:260] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:58.719273 24789 consensus_queue.cc:260] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:58.719386 24788 raft_consensus.cc:388] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:58.719434 24789 raft_consensus.cc:388] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:58.719480 24788 raft_consensus.cc:482] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:58.719521 24789 raft_consensus.cc:482] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:58.719604 24788 raft_consensus.cc:3037] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:58.719645 24789 raft_consensus.cc:3037] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:58.721789 24788 raft_consensus.cc:504] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:58.721881 24789 raft_consensus.cc:504] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:58.722025 24788 leader_election.cc:302] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3de510fd6a8a47a6acc24acba19b2364; no voters:
I20240502 08:16:58.722173 24789 leader_election.cc:302] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd7173a12d4a43b7bc728c0d99db5384; no voters:
I20240502 08:16:58.722306 24778 raft_consensus.cc:2781] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:58.722393 24789 leader_election.cc:288] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:58.722472 24777 raft_consensus.cc:2781] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:58.722574 24778 raft_consensus.cc:686] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Becoming Leader. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Running, Role: LEADER
I20240502 08:16:58.722885 24778 consensus_queue.cc:237] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } }
I20240502 08:16:58.723688 24788 leader_election.cc:288] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:58.724493 24788 ts_tablet_manager.cc:1398] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20240502 08:16:58.725783 23545 catalog_manager.cc:5484] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:58.731029 24789 ts_tablet_manager.cc:1398] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384: Time spent starting tablet: real 0.013s user 0.003s sys 0.008s
I20240502 08:16:58.731842 24777 raft_consensus.cc:686] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Becoming Leader. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Running, Role: LEADER
I20240502 08:16:58.732268 24789 tablet_bootstrap.cc:492] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384: Bootstrap starting.
I20240502 08:16:58.732152 24777 consensus_queue.cc:237] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:58.734225 24789 tablet_bootstrap.cc:654] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.735502 23549 catalog_manager.cc:5484] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:58.736217 24789 tablet_bootstrap.cc:492] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384: No bootstrap required, opened a new log
I20240502 08:16:58.736385 24789 ts_tablet_manager.cc:1367] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 08:16:58.737006 24789 raft_consensus.cc:348] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:58.737152 24789 raft_consensus.cc:374] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.737223 24789 raft_consensus.cc:729] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.737480 24789 consensus_queue.cc:260] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:58.737651 24789 raft_consensus.cc:388] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:58.737751 24789 raft_consensus.cc:482] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:58.737847 24789 raft_consensus.cc:3037] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:58.740388 24789 raft_consensus.cc:504] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:58.740587 24789 leader_election.cc:302] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd7173a12d4a43b7bc728c0d99db5384; no voters:
I20240502 08:16:58.740799 24789 leader_election.cc:288] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:58.740847 24777 raft_consensus.cc:2781] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:58.741024 24777 raft_consensus.cc:686] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Becoming Leader. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Running, Role: LEADER
I20240502 08:16:58.741251 24777 consensus_queue.cc:237] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } }
I20240502 08:16:58.741357 24789 ts_tablet_manager.cc:1398] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20240502 08:16:58.743148 23549 catalog_manager.cc:5484] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } health_report { overall_health: HEALTHY } } }
W20240502 08:16:59.082340 24798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:59.082679 24798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:59.090492 24798 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 122.265 ms
time per row: 0.0611324 ms
I20240502 08:16:59.354205 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.3:45649
--local_ip_for_outbound_sockets=127.20.207.3
--tserver_master_addrs=127.20.207.62:36471
--webserver_port=37435
--webserver_interface=127.20.207.3
--builtin_ntp_servers=127.20.207.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:16:59.693694 24838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:59.693990 24838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:59.694142 24838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:59.701654 24838 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:59.701789 24838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:59.701963 24838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.3
I20240502 08:16:59.711506 24838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.3:45649
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.207.3
--webserver_port=37435
--tserver_master_addrs=127.20.207.62:36471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.3
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:59.712886 24838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:59.714686 24838 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:59.728334 24844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:59.728780 24846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:59.729503 24843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:59.730419 24838 server_base.cc:1034] running on GCE node
I20240502 08:16:59.731096 24838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:59.732136 24838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:59.733363 24838 hybrid_clock.cc:648] HybridClock initialized: now 1714637819733323 us; error 33 us; skew 500 ppm
I20240502 08:16:59.733672 24838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:59.735993 24838 webserver.cc:460] Webserver started at http://127.20.207.3:37435/ using document root <none> and password file <none>
I20240502 08:16:59.736766 24838 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:59.736893 24838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:59.742478 24838 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:59.745486 24853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:59.746961 24838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:59.747125 24838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "39c4a2d0ccc94109b6bd2cac1792003f"
format_stamp: "Formatted at 2024-05-02 08:16:49 on dist-test-slave-169q"
I20240502 08:16:59.747982 24838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:59.778311 24838 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:59.779245 24838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:59.779556 24838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:59.780587 24838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:59.782196 24838 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:59.782292 24838 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:59.782420 24838 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:59.782474 24838 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:59.814795 24838 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.3:45649
I20240502 08:16:59.814842 24965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.3:45649 every 8 connection(s)
I20240502 08:16:59.816334 24838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:16:59.824465 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 24838
I20240502 08:16:59.825219 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.4:43335
--local_ip_for_outbound_sockets=127.20.207.4
--tserver_master_addrs=127.20.207.62:36471
--webserver_port=33821
--webserver_interface=127.20.207.4
--builtin_ntp_servers=127.20.207.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:16:59.832638 24966 heartbeater.cc:347] Connected to a master server at 127.20.207.62:36471
I20240502 08:16:59.832993 24966 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:59.834084 24966 heartbeater.cc:511] Master 127.20.207.62:36471 requested a full tablet report, sending...
I20240502 08:16:59.835919 23545 ts_manager.cc:194] Re-registered known tserver with Master: 39c4a2d0ccc94109b6bd2cac1792003f (127.20.207.3:45649)
I20240502 08:16:59.836787 23545 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.3:32923
W20240502 08:17:00.161795 24970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:00.162151 24970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:00.162303 24970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:00.170783 24970 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:00.170946 24970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:00.171167 24970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.4
I20240502 08:17:00.180388 24970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.4:43335
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.207.4
--webserver_port=33821
--tserver_master_addrs=127.20.207.62:36471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.4
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:00.181777 24970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:00.183571 24970 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:00.195802 24978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:00.196296 24975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:00.196964 24976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:00.197830 24970 server_base.cc:1034] running on GCE node
I20240502 08:17:00.198489 24970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:00.199438 24970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:00.200634 24970 hybrid_clock.cc:648] HybridClock initialized: now 1714637820200583 us; error 46 us; skew 500 ppm
I20240502 08:17:00.200947 24970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:00.203262 24970 webserver.cc:460] Webserver started at http://127.20.207.4:33821/ using document root <none> and password file <none>
I20240502 08:17:00.204034 24970 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:00.204169 24970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:00.209551 24970 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20240502 08:17:00.212419 24984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:00.213763 24970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 08:17:00.213925 24970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "67e133ffa16d4a7a9ce63f934ab783b0"
format_stamp: "Formatted at 2024-05-02 08:16:50 on dist-test-slave-169q"
I20240502 08:17:00.214793 24970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:00.229282 24970 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:00.230168 24970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:00.230471 24970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:00.231434 24970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:00.233038 24970 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:00.233117 24970 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:00.233238 24970 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:00.233294 24970 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:00.266736 25096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.4:43335 every 8 connection(s)
I20240502 08:17:00.266736 24970 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.4:43335
I20240502 08:17:00.268204 24970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:17:00.277196 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 24970
I20240502 08:17:00.277875 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.5:44257
--local_ip_for_outbound_sockets=127.20.207.5
--tserver_master_addrs=127.20.207.62:36471
--webserver_port=40265
--webserver_interface=127.20.207.5
--builtin_ntp_servers=127.20.207.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:00.280860 25097 heartbeater.cc:347] Connected to a master server at 127.20.207.62:36471
I20240502 08:17:00.281175 25097 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:00.282088 25097 heartbeater.cc:511] Master 127.20.207.62:36471 requested a full tablet report, sending...
I20240502 08:17:00.283627 23545 ts_manager.cc:194] Re-registered known tserver with Master: 67e133ffa16d4a7a9ce63f934ab783b0 (127.20.207.4:43335)
I20240502 08:17:00.284435 23545 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.4:36013
W20240502 08:17:00.606446 25100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:00.606763 25100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:00.606894 25100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:00.614416 25100 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:00.614554 25100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:00.614717 25100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.5
I20240502 08:17:00.624104 25100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.5:44257
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.207.5
--webserver_port=40265
--tserver_master_addrs=127.20.207.62:36471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.5
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:00.625478 25100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:00.627359 25100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:00.640327 25105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:00.640877 25106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:00.641778 25108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:00.642685 25100 server_base.cc:1034] running on GCE node
I20240502 08:17:00.643312 25100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:00.644528 25100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:00.645759 25100 hybrid_clock.cc:648] HybridClock initialized: now 1714637820645707 us; error 46 us; skew 500 ppm
I20240502 08:17:00.646126 25100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:00.648540 25100 webserver.cc:460] Webserver started at http://127.20.207.5:40265/ using document root <none> and password file <none>
I20240502 08:17:00.649260 25100 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:00.649376 25100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:00.654995 25100 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:00.657964 25114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:00.659471 25100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 08:17:00.659626 25100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "962f20ef8dea4cac91c4bf7603a655c0"
format_stamp: "Formatted at 2024-05-02 08:16:50 on dist-test-slave-169q"
I20240502 08:17:00.660475 25100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:00.689893 25100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:00.690855 25100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:00.691147 25100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:00.692145 25100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:00.693708 25100 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:00.693783 25100 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:00.693882 25100 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:00.693943 25100 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:00.725318 25100 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.5:44257
I20240502 08:17:00.725374 25226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.5:44257 every 8 connection(s)
I20240502 08:17:00.726899 25100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:17:00.728066 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 25100
I20240502 08:17:00.728816 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.6:42387
--local_ip_for_outbound_sockets=127.20.207.6
--tserver_master_addrs=127.20.207.62:36471
--webserver_port=43415
--webserver_interface=127.20.207.6
--builtin_ntp_servers=127.20.207.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:00.742836 25227 heartbeater.cc:347] Connected to a master server at 127.20.207.62:36471
I20240502 08:17:00.743180 25227 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:00.744132 25227 heartbeater.cc:511] Master 127.20.207.62:36471 requested a full tablet report, sending...
I20240502 08:17:00.745693 23548 ts_manager.cc:194] Re-registered known tserver with Master: 962f20ef8dea4cac91c4bf7603a655c0 (127.20.207.5:44257)
I20240502 08:17:00.746704 23548 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.5:47781
I20240502 08:17:00.840348 24966 heartbeater.cc:503] Master 127.20.207.62:36471 was elected leader, sending a full tablet report...
W20240502 08:17:01.061645 25230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:01.061968 25230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:01.062116 25230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:01.069662 25230 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:01.069792 25230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:01.069903 25230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.6
I20240502 08:17:01.079080 25230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.6:42387
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.207.6
--webserver_port=43415
--tserver_master_addrs=127.20.207.62:36471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.6
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:01.080395 25230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:01.082178 25230 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:01.094367 25237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:01.095046 25235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:01.095736 25239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:01.096509 25230 server_base.cc:1034] running on GCE node
I20240502 08:17:01.097092 25230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:01.098052 25230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:01.099242 25230 hybrid_clock.cc:648] HybridClock initialized: now 1714637821099194 us; error 53 us; skew 500 ppm
I20240502 08:17:01.099560 25230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:01.106492 25230 webserver.cc:460] Webserver started at http://127.20.207.6:43415/ using document root <none> and password file <none>
I20240502 08:17:01.107285 25230 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:01.107409 25230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:01.112664 25230 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 08:17:01.115592 25245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:01.116971 25230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:01.117128 25230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4"
format_stamp: "Formatted at 2024-05-02 08:16:51 on dist-test-slave-169q"
I20240502 08:17:01.117981 25230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:01.138162 25230 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:01.139083 25230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:01.139381 25230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:01.140372 25230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:01.141929 25230 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:01.142036 25230 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:01.142163 25230 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:01.142215 25230 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:01.176363 25230 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.6:42387
I20240502 08:17:01.176399 25358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.6:42387 every 8 connection(s)
I20240502 08:17:01.177924 25230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:17:01.179497 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 25230
I20240502 08:17:01.180222 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.7:39999
--local_ip_for_outbound_sockets=127.20.207.7
--tserver_master_addrs=127.20.207.62:36471
--webserver_port=32859
--webserver_interface=127.20.207.7
--builtin_ntp_servers=127.20.207.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:01.193110 25359 heartbeater.cc:347] Connected to a master server at 127.20.207.62:36471
I20240502 08:17:01.193440 25359 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:01.194325 25359 heartbeater.cc:511] Master 127.20.207.62:36471 requested a full tablet report, sending...
I20240502 08:17:01.195812 23548 ts_manager.cc:194] Re-registered known tserver with Master: 4b439f9c3d53400e8e2dc7b5c0a724b4 (127.20.207.6:42387)
I20240502 08:17:01.196617 23548 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.6:36221
I20240502 08:17:01.287079 25097 heartbeater.cc:503] Master 127.20.207.62:36471 was elected leader, sending a full tablet report...
W20240502 08:17:01.529258 25362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:01.529580 25362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:01.529675 25362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:01.537501 25362 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:01.537637 25362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:01.537794 25362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.7
I20240502 08:17:01.547130 25362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.7:39999
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.207.7
--webserver_port=32859
--tserver_master_addrs=127.20.207.62:36471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.7
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:01.548470 25362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:01.550359 25362 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:01.562916 25367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:01.563565 25368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:01.564358 25370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:01.565256 25362 server_base.cc:1034] running on GCE node
I20240502 08:17:01.565918 25362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:01.567050 25362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:01.568259 25362 hybrid_clock.cc:648] HybridClock initialized: now 1714637821568197 us; error 45 us; skew 500 ppm
I20240502 08:17:01.568578 25362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:01.570917 25362 webserver.cc:460] Webserver started at http://127.20.207.7:32859/ using document root <none> and password file <none>
I20240502 08:17:01.571677 25362 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:01.571806 25362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:01.577308 25362 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:17:01.580276 25376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:01.581686 25362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:01.581871 25362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "cfb4a4db281241f5940203fe907dea50"
format_stamp: "Formatted at 2024-05-02 08:16:51 on dist-test-slave-169q"
I20240502 08:17:01.582803 25362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:01.596969 25362 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:01.597811 25362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:01.598384 25362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:01.599450 25362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:01.601114 25362 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:01.601200 25362 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:01.601363 25362 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:01.601418 25362 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:01.634543 25362 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.7:39999
I20240502 08:17:01.634591 25489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.7:39999 every 8 connection(s)
I20240502 08:17:01.635996 25362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 08:17:01.642638 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 25362
I20240502 08:17:01.649799 25490 heartbeater.cc:347] Connected to a master server at 127.20.207.62:36471
I20240502 08:17:01.650157 25490 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:01.650993 25490 heartbeater.cc:511] Master 127.20.207.62:36471 requested a full tablet report, sending...
I20240502 08:17:01.652515 23548 ts_manager.cc:194] Re-registered known tserver with Master: cfb4a4db281241f5940203fe907dea50 (127.20.207.7:39999)
I20240502 08:17:01.653383 23548 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.7:40599
I20240502 08:17:01.749179 25227 heartbeater.cc:503] Master 127.20.207.62:36471 was elected leader, sending a full tablet report...
I20240502 08:17:02.154860 23675 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:17:02.163645 24901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:02.173615 25162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:02.174665 25293 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:02.193902 25425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:02.199167 25359 heartbeater.cc:503] Master 127.20.207.62:36471 was elected leader, sending a full tablet report...
I20240502 08:17:02.216429 25032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:02.242627 23808 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:17:02.424116 24901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:02.425705 25293 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:02.436553 23675 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:17:02.444691 25032 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:02.450862 25162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:02.455283 23808 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:17:02.468009 25425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:02.545573 23696 consensus_queue.cc:237] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: true } }
I20240502 08:17:02.548171 25582 raft_consensus.cc:2932] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cfb4a4db281241f5940203fe907dea50 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: true } } }
I20240502 08:17:02.555160 23545 catalog_manager.cc:5484] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: config changed from index -1 to 4, NON_VOTER cfb4a4db281241f5940203fe907dea50 (127.20.207.7) added. New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:02.558943 23632 consensus_peers.cc:485] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 -> Peer cfb4a4db281241f5940203fe907dea50 (127.20.207.7:39999): Couldn't send request to peer cfb4a4db281241f5940203fe907dea50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c26c8082287402daf70ee9be5a4fb7f. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.570663 23828 consensus_queue.cc:237] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: true } }
I20240502 08:17:02.573287 25595 raft_consensus.cc:2932] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 39c4a2d0ccc94109b6bd2cac1792003f (127.20.207.3) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: true } } }
I20240502 08:17:02.579192 23545 catalog_manager.cc:5484] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: config changed from index -1 to 4, NON_VOTER 39c4a2d0ccc94109b6bd2cac1792003f (127.20.207.3) added. New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:02.584267 23761 consensus_peers.cc:485] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 -> Peer 39c4a2d0ccc94109b6bd2cac1792003f (127.20.207.3:45649): Couldn't send request to peer 39c4a2d0ccc94109b6bd2cac1792003f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce947f6507ea457da00437f718444e5b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.590404 23696 consensus_queue.cc:237] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: true } }
I20240502 08:17:02.592188 25582 raft_consensus.cc:2932] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 67e133ffa16d4a7a9ce63f934ab783b0 (127.20.207.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: true } } }
I20240502 08:17:02.597231 23545 catalog_manager.cc:5484] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: config changed from index -1 to 4, NON_VOTER 67e133ffa16d4a7a9ce63f934ab783b0 (127.20.207.4) added. New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:02.602273 23631 consensus_peers.cc:485] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 -> Peer 67e133ffa16d4a7a9ce63f934ab783b0 (127.20.207.4:43335): Couldn't send request to peer 67e133ffa16d4a7a9ce63f934ab783b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7df5ab24ba3c42769dd4640220641996. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.613888 23696 consensus_queue.cc:237] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: true } }
I20240502 08:17:02.615592 25582 raft_consensus.cc:2932] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 962f20ef8dea4cac91c4bf7603a655c0 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: true } } }
I20240502 08:17:02.621047 23545 catalog_manager.cc:5484] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: config changed from index -1 to 4, NON_VOTER 962f20ef8dea4cac91c4bf7603a655c0 (127.20.207.5) added. New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:02.623703 23630 consensus_peers.cc:485] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 -> Peer 962f20ef8dea4cac91c4bf7603a655c0 (127.20.207.5:44257): Couldn't send request to peer 962f20ef8dea4cac91c4bf7603a655c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db6049ba7a5640d4ac4dae656b2f3667. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.632638 23828 consensus_queue.cc:237] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: true } }
I20240502 08:17:02.634653 25595 raft_consensus.cc:2932] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4b439f9c3d53400e8e2dc7b5c0a724b4 (127.20.207.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: true } } }
I20240502 08:17:02.639935 23545 catalog_manager.cc:5484] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: config changed from index -1 to 4, NON_VOTER 4b439f9c3d53400e8e2dc7b5c0a724b4 (127.20.207.6) added. New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:02.642741 23762 consensus_peers.cc:485] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 -> Peer 4b439f9c3d53400e8e2dc7b5c0a724b4 (127.20.207.6:42387): Couldn't send request to peer 4b439f9c3d53400e8e2dc7b5c0a724b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f214664b085a4ac0a10484acbd525142. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.653720 23828 consensus_queue.cc:237] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: true } }
I20240502 08:17:02.655606 25595 raft_consensus.cc:2932] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cfb4a4db281241f5940203fe907dea50 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: true } } }
I20240502 08:17:02.656361 25490 heartbeater.cc:503] Master 127.20.207.62:36471 was elected leader, sending a full tablet report...
I20240502 08:17:02.661475 23546 catalog_manager.cc:5484] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: config changed from index -1 to 4, NON_VOTER cfb4a4db281241f5940203fe907dea50 (127.20.207.7) added. New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:02.662271 23763 consensus_peers.cc:485] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 -> Peer cfb4a4db281241f5940203fe907dea50 (127.20.207.7:39999): Couldn't send request to peer cfb4a4db281241f5940203fe907dea50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c0157c3916c4012a6b8c4d38df7a342. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.671425 23828 consensus_queue.cc:237] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: true } }
I20240502 08:17:02.673055 25595 raft_consensus.cc:2932] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 39c4a2d0ccc94109b6bd2cac1792003f (127.20.207.3) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: true } } }
W20240502 08:17:02.673514 23761 consensus_peers.cc:485] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 -> Peer 39c4a2d0ccc94109b6bd2cac1792003f (127.20.207.3:45649): Couldn't send request to peer 39c4a2d0ccc94109b6bd2cac1792003f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb067e2ff17740958059cfef1e0d0f9d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.677743 23546 catalog_manager.cc:5484] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: config changed from index -1 to 4, NON_VOTER 39c4a2d0ccc94109b6bd2cac1792003f (127.20.207.3) added. New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:02.691301 23828 consensus_queue.cc:237] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: true } }
I20240502 08:17:02.693037 25627 raft_consensus.cc:2932] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 67e133ffa16d4a7a9ce63f934ab783b0 (127.20.207.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: true } } }
I20240502 08:17:02.698686 23546 catalog_manager.cc:5484] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: config changed from index -1 to 4, NON_VOTER 67e133ffa16d4a7a9ce63f934ab783b0 (127.20.207.4) added. New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:02.700137 23762 consensus_peers.cc:485] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 -> Peer 67e133ffa16d4a7a9ce63f934ab783b0 (127.20.207.4:43335): Couldn't send request to peer 67e133ffa16d4a7a9ce63f934ab783b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9d6f55ce8f547e395d686838dbb7bd2. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.710844 23696 consensus_queue.cc:237] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: true } }
I20240502 08:17:02.712410 25582 raft_consensus.cc:2932] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 962f20ef8dea4cac91c4bf7603a655c0 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: true } } }
W20240502 08:17:02.712883 23630 consensus_peers.cc:485] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 -> Peer 962f20ef8dea4cac91c4bf7603a655c0 (127.20.207.5:44257): Couldn't send request to peer 962f20ef8dea4cac91c4bf7603a655c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8096d61335f8493594e55720d47489bd. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.717319 23545 catalog_manager.cc:5484] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: config changed from index -1 to 4, NON_VOTER 962f20ef8dea4cac91c4bf7603a655c0 (127.20.207.5) added. New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:02.727540 23696 consensus_queue.cc:237] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: true } }
I20240502 08:17:02.729377 25582 raft_consensus.cc:2932] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4b439f9c3d53400e8e2dc7b5c0a724b4 (127.20.207.6) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: true } } }
W20240502 08:17:02.735047 23631 consensus_peers.cc:485] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 -> Peer 4b439f9c3d53400e8e2dc7b5c0a724b4 (127.20.207.6:42387): Couldn't send request to peer 4b439f9c3d53400e8e2dc7b5c0a724b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eff43ec5b0ea4198ae23d57f2c4cb2c9. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.736428 23545 catalog_manager.cc:5484] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: config changed from index -1 to 2, NON_VOTER 4b439f9c3d53400e8e2dc7b5c0a724b4 (127.20.207.6) added. New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:02.963371 25698 ts_tablet_manager.cc:902] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f: Initiating tablet copy from peer dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263)
I20240502 08:17:02.965181 25698 tablet_copy_client.cc:308] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.2:35263
I20240502 08:17:02.971856 23849 tablet_copy_service.cc:140] P dd7173a12d4a43b7bc728c0d99db5384: Received BeginTabletCopySession request for tablet ce947f6507ea457da00437f718444e5b from peer 39c4a2d0ccc94109b6bd2cac1792003f ({username='slave'} at 127.20.207.3:39623)
I20240502 08:17:02.972100 23849 tablet_copy_service.cc:161] P dd7173a12d4a43b7bc728c0d99db5384: Beginning new tablet copy session on tablet ce947f6507ea457da00437f718444e5b from peer 39c4a2d0ccc94109b6bd2cac1792003f at {username='slave'} at 127.20.207.3:39623: session id = 39c4a2d0ccc94109b6bd2cac1792003f-ce947f6507ea457da00437f718444e5b
I20240502 08:17:02.974838 23849 tablet_copy_source_session.cc:205] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:02.979534 25698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce947f6507ea457da00437f718444e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.990283 25698 tablet_copy_client.cc:783] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:02.990628 25698 tablet_copy_client.cc:647] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:02.993484 25698 tablet_copy_client.cc:523] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:02.996954 25698 tablet_bootstrap.cc:492] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f: Bootstrap starting.
I20240502 08:17:03.006916 25698 log.cc:824] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:03.009413 25705 ts_tablet_manager.cc:902] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0: Initiating tablet copy from peer 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519)
I20240502 08:17:03.011442 25705 tablet_copy_client.cc:308] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:41519
I20240502 08:17:03.016048 25711 ts_tablet_manager.cc:902] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4: Initiating tablet copy from peer dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263)
I20240502 08:17:03.019169 25711 tablet_copy_client.cc:308] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.2:35263
I20240502 08:17:03.019209 23716 tablet_copy_service.cc:140] P 3de510fd6a8a47a6acc24acba19b2364: Received BeginTabletCopySession request for tablet db6049ba7a5640d4ac4dae656b2f3667 from peer 962f20ef8dea4cac91c4bf7603a655c0 ({username='slave'} at 127.20.207.5:49045)
I20240502 08:17:03.019481 23716 tablet_copy_service.cc:161] P 3de510fd6a8a47a6acc24acba19b2364: Beginning new tablet copy session on tablet db6049ba7a5640d4ac4dae656b2f3667 from peer 962f20ef8dea4cac91c4bf7603a655c0 at {username='slave'} at 127.20.207.5:49045: session id = 962f20ef8dea4cac91c4bf7603a655c0-db6049ba7a5640d4ac4dae656b2f3667
I20240502 08:17:03.022933 23716 tablet_copy_source_session.cc:205] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:03.027560 23849 tablet_copy_service.cc:140] P dd7173a12d4a43b7bc728c0d99db5384: Received BeginTabletCopySession request for tablet f214664b085a4ac0a10484acbd525142 from peer 4b439f9c3d53400e8e2dc7b5c0a724b4 ({username='slave'} at 127.20.207.6:38227)
I20240502 08:17:03.027766 23849 tablet_copy_service.cc:161] P dd7173a12d4a43b7bc728c0d99db5384: Beginning new tablet copy session on tablet f214664b085a4ac0a10484acbd525142 from peer 4b439f9c3d53400e8e2dc7b5c0a724b4 at {username='slave'} at 127.20.207.6:38227: session id = 4b439f9c3d53400e8e2dc7b5c0a724b4-f214664b085a4ac0a10484acbd525142
I20240502 08:17:03.027972 25705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db6049ba7a5640d4ac4dae656b2f3667. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.030656 23849 tablet_copy_source_session.cc:205] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:03.042773 25705 tablet_copy_client.cc:783] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:03.043258 25705 tablet_copy_client.cc:647] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:03.045121 25711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f214664b085a4ac0a10484acbd525142. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.046623 25705 tablet_copy_client.cc:523] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:03.052990 25705 tablet_bootstrap.cc:492] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0: Bootstrap starting.
I20240502 08:17:03.061398 25698 tablet_bootstrap.cc:492] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:03.062079 25698 tablet_bootstrap.cc:492] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f: Bootstrap complete.
I20240502 08:17:03.062659 25698 ts_tablet_manager.cc:1367] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f: Time spent bootstrapping tablet: real 0.066s user 0.062s sys 0.004s
I20240502 08:17:03.063419 25720 ts_tablet_manager.cc:902] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50: Initiating tablet copy from peer 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519)
I20240502 08:17:03.067502 25705 log.cc:824] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:03.069341 25698 raft_consensus.cc:348] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: true } }
I20240502 08:17:03.069800 25698 raft_consensus.cc:729] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 39c4a2d0ccc94109b6bd2cac1792003f, State: Initialized, Role: LEARNER
I20240502 08:17:03.070699 25698 consensus_queue.cc:260] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: true } }
I20240502 08:17:03.077611 25711 tablet_copy_client.cc:783] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:03.078137 25711 tablet_copy_client.cc:647] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:03.082129 25711 tablet_copy_client.cc:523] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:03.090963 25720 tablet_copy_client.cc:308] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:41519
I20240502 08:17:03.092784 25727 ts_tablet_manager.cc:902] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50: Initiating tablet copy from peer dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263)
I20240502 08:17:03.096913 25711 tablet_bootstrap.cc:492] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4: Bootstrap starting.
I20240502 08:17:03.101678 23716 tablet_copy_service.cc:140] P 3de510fd6a8a47a6acc24acba19b2364: Received BeginTabletCopySession request for tablet 1c26c8082287402daf70ee9be5a4fb7f from peer cfb4a4db281241f5940203fe907dea50 ({username='slave'} at 127.20.207.7:58099)
I20240502 08:17:03.101892 23716 tablet_copy_service.cc:161] P 3de510fd6a8a47a6acc24acba19b2364: Beginning new tablet copy session on tablet 1c26c8082287402daf70ee9be5a4fb7f from peer cfb4a4db281241f5940203fe907dea50 at {username='slave'} at 127.20.207.7:58099: session id = cfb4a4db281241f5940203fe907dea50-1c26c8082287402daf70ee9be5a4fb7f
I20240502 08:17:03.104094 25698 ts_tablet_manager.cc:1398] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f: Time spent starting tablet: real 0.041s user 0.023s sys 0.016s
I20240502 08:17:03.105202 23716 tablet_copy_source_session.cc:205] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:03.105230 23849 tablet_copy_service.cc:342] P dd7173a12d4a43b7bc728c0d99db5384: Request end of tablet copy session 39c4a2d0ccc94109b6bd2cac1792003f-ce947f6507ea457da00437f718444e5b received from {username='slave'} at 127.20.207.3:39623
I20240502 08:17:03.105432 23849 tablet_copy_service.cc:433] P dd7173a12d4a43b7bc728c0d99db5384: ending tablet copy session 39c4a2d0ccc94109b6bd2cac1792003f-ce947f6507ea457da00437f718444e5b on tablet ce947f6507ea457da00437f718444e5b with peer 39c4a2d0ccc94109b6bd2cac1792003f
I20240502 08:17:03.112177 25720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c26c8082287402daf70ee9be5a4fb7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.128512 25711 log.cc:824] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:03.129485 25720 tablet_copy_client.cc:783] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:03.129953 25720 tablet_copy_client.cc:647] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:03.132905 25720 tablet_copy_client.cc:523] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:03.137696 25720 tablet_bootstrap.cc:492] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50: Bootstrap starting.
I20240502 08:17:03.141526 25727 tablet_copy_client.cc:308] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.2:35263
I20240502 08:17:03.143576 25705 tablet_bootstrap.cc:492] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:03.143596 25739 ts_tablet_manager.cc:902] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0: Initiating tablet copy from peer 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519)
I20240502 08:17:03.144358 25705 tablet_bootstrap.cc:492] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0: Bootstrap complete.
I20240502 08:17:03.144892 25705 ts_tablet_manager.cc:1367] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0: Time spent bootstrapping tablet: real 0.092s user 0.064s sys 0.004s
I20240502 08:17:03.150686 25698 ts_tablet_manager.cc:902] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f: Initiating tablet copy from peer dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263)
I20240502 08:17:03.152194 23849 tablet_copy_service.cc:140] P dd7173a12d4a43b7bc728c0d99db5384: Received BeginTabletCopySession request for tablet 2c0157c3916c4012a6b8c4d38df7a342 from peer cfb4a4db281241f5940203fe907dea50 ({username='slave'} at 127.20.207.7:34875)
I20240502 08:17:03.152406 23849 tablet_copy_service.cc:161] P dd7173a12d4a43b7bc728c0d99db5384: Beginning new tablet copy session on tablet 2c0157c3916c4012a6b8c4d38df7a342 from peer cfb4a4db281241f5940203fe907dea50 at {username='slave'} at 127.20.207.7:34875: session id = cfb4a4db281241f5940203fe907dea50-2c0157c3916c4012a6b8c4d38df7a342
I20240502 08:17:03.156165 25720 log.cc:824] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:03.156551 23849 tablet_copy_source_session.cc:205] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:03.158349 25727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c0157c3916c4012a6b8c4d38df7a342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.159929 25705 raft_consensus.cc:348] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: true } }
I20240502 08:17:03.161839 25705 raft_consensus.cc:729] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 962f20ef8dea4cac91c4bf7603a655c0, State: Initialized, Role: LEARNER
I20240502 08:17:03.162691 25705 consensus_queue.cc:260] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: true } }
I20240502 08:17:03.166131 25727 tablet_copy_client.cc:783] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:03.166383 25727 tablet_copy_client.cc:647] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:03.168112 25727 tablet_copy_client.cc:523] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:03.170313 25739 tablet_copy_client.cc:308] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:41519
I20240502 08:17:03.171686 25727 tablet_bootstrap.cc:492] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50: Bootstrap starting.
I20240502 08:17:03.184149 25705 ts_tablet_manager.cc:1398] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0: Time spent starting tablet: real 0.039s user 0.013s sys 0.004s
I20240502 08:17:03.187115 23716 tablet_copy_service.cc:342] P 3de510fd6a8a47a6acc24acba19b2364: Request end of tablet copy session 962f20ef8dea4cac91c4bf7603a655c0-db6049ba7a5640d4ac4dae656b2f3667 received from {username='slave'} at 127.20.207.5:49045
I20240502 08:17:03.187359 23716 tablet_copy_service.cc:433] P 3de510fd6a8a47a6acc24acba19b2364: ending tablet copy session 962f20ef8dea4cac91c4bf7603a655c0-db6049ba7a5640d4ac4dae656b2f3667 on tablet db6049ba7a5640d4ac4dae656b2f3667 with peer 962f20ef8dea4cac91c4bf7603a655c0
I20240502 08:17:03.205960 25711 tablet_bootstrap.cc:492] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:03.206624 25711 tablet_bootstrap.cc:492] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4: Bootstrap complete.
I20240502 08:17:03.206846 25698 tablet_copy_client.cc:308] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.2:35263
I20240502 08:17:03.207141 25711 ts_tablet_manager.cc:1367] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4: Time spent bootstrapping tablet: real 0.111s user 0.060s sys 0.011s
I20240502 08:17:03.207782 23849 tablet_copy_service.cc:140] P dd7173a12d4a43b7bc728c0d99db5384: Received BeginTabletCopySession request for tablet fb067e2ff17740958059cfef1e0d0f9d from peer 39c4a2d0ccc94109b6bd2cac1792003f ({username='slave'} at 127.20.207.3:39623)
I20240502 08:17:03.207976 23849 tablet_copy_service.cc:161] P dd7173a12d4a43b7bc728c0d99db5384: Beginning new tablet copy session on tablet fb067e2ff17740958059cfef1e0d0f9d from peer 39c4a2d0ccc94109b6bd2cac1792003f at {username='slave'} at 127.20.207.3:39623: session id = 39c4a2d0ccc94109b6bd2cac1792003f-fb067e2ff17740958059cfef1e0d0f9d
I20240502 08:17:03.210700 23716 tablet_copy_service.cc:140] P 3de510fd6a8a47a6acc24acba19b2364: Received BeginTabletCopySession request for tablet 7df5ab24ba3c42769dd4640220641996 from peer 67e133ffa16d4a7a9ce63f934ab783b0 ({username='slave'} at 127.20.207.4:35195)
I20240502 08:17:03.210848 23716 tablet_copy_service.cc:161] P 3de510fd6a8a47a6acc24acba19b2364: Beginning new tablet copy session on tablet 7df5ab24ba3c42769dd4640220641996 from peer 67e133ffa16d4a7a9ce63f934ab783b0 at {username='slave'} at 127.20.207.4:35195: session id = 67e133ffa16d4a7a9ce63f934ab783b0-7df5ab24ba3c42769dd4640220641996
I20240502 08:17:03.210963 23849 tablet_copy_source_session.cc:205] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:03.212388 25698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb067e2ff17740958059cfef1e0d0f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.213536 23716 tablet_copy_source_session.cc:205] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:03.213841 25711 raft_consensus.cc:348] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: true } }
I20240502 08:17:03.214346 25711 raft_consensus.cc:729] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b439f9c3d53400e8e2dc7b5c0a724b4, State: Initialized, Role: LEARNER
I20240502 08:17:03.214926 25711 consensus_queue.cc:260] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: true } }
I20240502 08:17:03.218907 25739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df5ab24ba3c42769dd4640220641996. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.220780 25698 tablet_copy_client.cc:783] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:03.221055 25698 tablet_copy_client.cc:647] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:03.222846 25698 tablet_copy_client.cc:523] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:03.226708 25698 tablet_bootstrap.cc:492] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f: Bootstrap starting.
I20240502 08:17:03.235503 25711 ts_tablet_manager.cc:1398] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4: Time spent starting tablet: real 0.028s user 0.009s sys 0.000s
I20240502 08:17:03.236423 23849 tablet_copy_service.cc:342] P dd7173a12d4a43b7bc728c0d99db5384: Request end of tablet copy session 4b439f9c3d53400e8e2dc7b5c0a724b4-f214664b085a4ac0a10484acbd525142 received from {username='slave'} at 127.20.207.6:38227
I20240502 08:17:03.236558 23849 tablet_copy_service.cc:433] P dd7173a12d4a43b7bc728c0d99db5384: ending tablet copy session 4b439f9c3d53400e8e2dc7b5c0a724b4-f214664b085a4ac0a10484acbd525142 on tablet f214664b085a4ac0a10484acbd525142 with peer 4b439f9c3d53400e8e2dc7b5c0a724b4
I20240502 08:17:03.238250 25757 ts_tablet_manager.cc:902] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0: Initiating tablet copy from peer dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263)
I20240502 08:17:03.240298 25739 tablet_copy_client.cc:783] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:03.240672 25739 tablet_copy_client.cc:647] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:03.243444 25739 tablet_copy_client.cc:523] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:03.247948 25739 tablet_bootstrap.cc:492] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0: Bootstrap starting.
I20240502 08:17:03.253242 25757 tablet_copy_client.cc:308] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.2:35263
I20240502 08:17:03.263008 25705 ts_tablet_manager.cc:902] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0: Initiating tablet copy from peer 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519)
I20240502 08:17:03.265585 23849 tablet_copy_service.cc:140] P dd7173a12d4a43b7bc728c0d99db5384: Received BeginTabletCopySession request for tablet c9d6f55ce8f547e395d686838dbb7bd2 from peer 67e133ffa16d4a7a9ce63f934ab783b0 ({username='slave'} at 127.20.207.4:50257)
I20240502 08:17:03.265818 23849 tablet_copy_service.cc:161] P dd7173a12d4a43b7bc728c0d99db5384: Beginning new tablet copy session on tablet c9d6f55ce8f547e395d686838dbb7bd2 from peer 67e133ffa16d4a7a9ce63f934ab783b0 at {username='slave'} at 127.20.207.4:50257: session id = 67e133ffa16d4a7a9ce63f934ab783b0-c9d6f55ce8f547e395d686838dbb7bd2
I20240502 08:17:03.268991 23849 tablet_copy_source_session.cc:205] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:03.270519 25757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d6f55ce8f547e395d686838dbb7bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.277617 25757 tablet_copy_client.cc:783] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:03.277927 25757 tablet_copy_client.cc:647] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:03.280092 25757 tablet_copy_client.cc:523] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:03.281296 25739 log.cc:824] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:03.284386 25757 tablet_bootstrap.cc:492] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0: Bootstrap starting.
I20240502 08:17:03.304807 25705 tablet_copy_client.cc:308] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:41519
I20240502 08:17:03.305781 23716 tablet_copy_service.cc:140] P 3de510fd6a8a47a6acc24acba19b2364: Received BeginTabletCopySession request for tablet 8096d61335f8493594e55720d47489bd from peer 962f20ef8dea4cac91c4bf7603a655c0 ({username='slave'} at 127.20.207.5:49045)
I20240502 08:17:03.305987 23716 tablet_copy_service.cc:161] P 3de510fd6a8a47a6acc24acba19b2364: Beginning new tablet copy session on tablet 8096d61335f8493594e55720d47489bd from peer 962f20ef8dea4cac91c4bf7603a655c0 at {username='slave'} at 127.20.207.5:49045: session id = 962f20ef8dea4cac91c4bf7603a655c0-8096d61335f8493594e55720d47489bd
I20240502 08:17:03.308830 23716 tablet_copy_source_session.cc:205] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:03.310632 25705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8096d61335f8493594e55720d47489bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.323864 25705 tablet_copy_client.cc:783] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:03.324149 25705 tablet_copy_client.cc:647] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:03.325764 25705 tablet_copy_client.cc:523] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:03.327651 25727 tablet_bootstrap.cc:492] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:03.328197 25727 tablet_bootstrap.cc:492] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50: Bootstrap complete.
I20240502 08:17:03.328708 25727 ts_tablet_manager.cc:1367] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50: Time spent bootstrapping tablet: real 0.157s user 0.051s sys 0.004s
I20240502 08:17:03.333518 25705 tablet_bootstrap.cc:492] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0: Bootstrap starting.
I20240502 08:17:03.335167 25711 ts_tablet_manager.cc:902] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4: Initiating tablet copy from peer 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519)
I20240502 08:17:03.344811 25711 tablet_copy_client.cc:308] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:41519
I20240502 08:17:03.350601 25727 raft_consensus.cc:348] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: true } }
I20240502 08:17:03.354645 23716 tablet_copy_service.cc:140] P 3de510fd6a8a47a6acc24acba19b2364: Received BeginTabletCopySession request for tablet eff43ec5b0ea4198ae23d57f2c4cb2c9 from peer 4b439f9c3d53400e8e2dc7b5c0a724b4 ({username='slave'} at 127.20.207.6:37247)
I20240502 08:17:03.354873 23716 tablet_copy_service.cc:161] P 3de510fd6a8a47a6acc24acba19b2364: Beginning new tablet copy session on tablet eff43ec5b0ea4198ae23d57f2c4cb2c9 from peer 4b439f9c3d53400e8e2dc7b5c0a724b4 at {username='slave'} at 127.20.207.6:37247: session id = 4b439f9c3d53400e8e2dc7b5c0a724b4-eff43ec5b0ea4198ae23d57f2c4cb2c9
I20240502 08:17:03.357513 25727 raft_consensus.cc:729] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cfb4a4db281241f5940203fe907dea50, State: Initialized, Role: LEARNER
I20240502 08:17:03.358484 23716 tablet_copy_source_session.cc:205] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:03.359345 25727 consensus_queue.cc:260] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: true } }
I20240502 08:17:03.359944 25711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff43ec5b0ea4198ae23d57f2c4cb2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.359843 25720 tablet_bootstrap.cc:492] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:03.360491 25720 tablet_bootstrap.cc:492] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50: Bootstrap complete.
I20240502 08:17:03.361287 25720 ts_tablet_manager.cc:1367] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50: Time spent bootstrapping tablet: real 0.224s user 0.079s sys 0.004s
I20240502 08:17:03.362443 25720 raft_consensus.cc:348] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: true } }
I20240502 08:17:03.362728 25720 raft_consensus.cc:729] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cfb4a4db281241f5940203fe907dea50, State: Initialized, Role: LEARNER
I20240502 08:17:03.362988 25720 consensus_queue.cc:260] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: true } }
I20240502 08:17:03.363420 25698 tablet_bootstrap.cc:492] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:03.363898 25698 tablet_bootstrap.cc:492] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f: Bootstrap complete.
I20240502 08:17:03.364298 25698 ts_tablet_manager.cc:1367] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f: Time spent bootstrapping tablet: real 0.138s user 0.054s sys 0.000s
I20240502 08:17:03.365397 25698 raft_consensus.cc:348] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: true } }
I20240502 08:17:03.365638 25698 raft_consensus.cc:729] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 39c4a2d0ccc94109b6bd2cac1792003f, State: Initialized, Role: LEARNER
I20240502 08:17:03.365870 25698 consensus_queue.cc:260] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: true } }
I20240502 08:17:03.369109 25698 ts_tablet_manager.cc:1398] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:17:03.370271 25711 tablet_copy_client.cc:783] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:03.370607 25711 tablet_copy_client.cc:647] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:03.371970 23849 tablet_copy_service.cc:342] P dd7173a12d4a43b7bc728c0d99db5384: Request end of tablet copy session 39c4a2d0ccc94109b6bd2cac1792003f-fb067e2ff17740958059cfef1e0d0f9d received from {username='slave'} at 127.20.207.3:39623
I20240502 08:17:03.372128 23849 tablet_copy_service.cc:433] P dd7173a12d4a43b7bc728c0d99db5384: ending tablet copy session 39c4a2d0ccc94109b6bd2cac1792003f-fb067e2ff17740958059cfef1e0d0f9d on tablet fb067e2ff17740958059cfef1e0d0f9d with peer 39c4a2d0ccc94109b6bd2cac1792003f
I20240502 08:17:03.364601 25739 tablet_bootstrap.cc:492] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:03.372915 25711 tablet_copy_client.cc:523] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:03.373140 25739 tablet_bootstrap.cc:492] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0: Bootstrap complete.
I20240502 08:17:03.373674 25739 ts_tablet_manager.cc:1367] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0: Time spent bootstrapping tablet: real 0.126s user 0.066s sys 0.023s
I20240502 08:17:03.377449 25711 tablet_bootstrap.cc:492] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4: Bootstrap starting.
I20240502 08:17:03.380334 25727 ts_tablet_manager.cc:1398] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50: Time spent starting tablet: real 0.051s user 0.015s sys 0.004s
I20240502 08:17:03.380398 25739 raft_consensus.cc:348] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: true } }
I20240502 08:17:03.380874 25739 raft_consensus.cc:729] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 67e133ffa16d4a7a9ce63f934ab783b0, State: Initialized, Role: LEARNER
I20240502 08:17:03.381482 23849 tablet_copy_service.cc:342] P dd7173a12d4a43b7bc728c0d99db5384: Request end of tablet copy session cfb4a4db281241f5940203fe907dea50-2c0157c3916c4012a6b8c4d38df7a342 received from {username='slave'} at 127.20.207.7:34875
I20240502 08:17:03.381429 25739 consensus_queue.cc:260] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: true } }
I20240502 08:17:03.381642 23849 tablet_copy_service.cc:433] P dd7173a12d4a43b7bc728c0d99db5384: ending tablet copy session cfb4a4db281241f5940203fe907dea50-2c0157c3916c4012a6b8c4d38df7a342 on tablet 2c0157c3916c4012a6b8c4d38df7a342 with peer cfb4a4db281241f5940203fe907dea50
I20240502 08:17:03.383853 25720 ts_tablet_manager.cc:1398] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50: Time spent starting tablet: real 0.022s user 0.002s sys 0.007s
I20240502 08:17:03.384683 23716 tablet_copy_service.cc:342] P 3de510fd6a8a47a6acc24acba19b2364: Request end of tablet copy session cfb4a4db281241f5940203fe907dea50-1c26c8082287402daf70ee9be5a4fb7f received from {username='slave'} at 127.20.207.7:58099
I20240502 08:17:03.384868 23716 tablet_copy_service.cc:433] P 3de510fd6a8a47a6acc24acba19b2364: ending tablet copy session cfb4a4db281241f5940203fe907dea50-1c26c8082287402daf70ee9be5a4fb7f on tablet 1c26c8082287402daf70ee9be5a4fb7f with peer cfb4a4db281241f5940203fe907dea50
I20240502 08:17:03.387755 25739 ts_tablet_manager.cc:1398] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20240502 08:17:03.388763 23716 tablet_copy_service.cc:342] P 3de510fd6a8a47a6acc24acba19b2364: Request end of tablet copy session 67e133ffa16d4a7a9ce63f934ab783b0-7df5ab24ba3c42769dd4640220641996 received from {username='slave'} at 127.20.207.4:35195
I20240502 08:17:03.388931 23716 tablet_copy_service.cc:433] P 3de510fd6a8a47a6acc24acba19b2364: ending tablet copy session 67e133ffa16d4a7a9ce63f934ab783b0-7df5ab24ba3c42769dd4640220641996 on tablet 7df5ab24ba3c42769dd4640220641996 with peer 67e133ffa16d4a7a9ce63f934ab783b0
I20240502 08:17:03.396994 25711 tablet_bootstrap.cc:492] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:03.397437 25711 tablet_bootstrap.cc:492] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4: Bootstrap complete.
I20240502 08:17:03.397691 25711 ts_tablet_manager.cc:1367] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.004s
I20240502 08:17:03.401471 25711 raft_consensus.cc:348] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: true } }
I20240502 08:17:03.401708 25711 raft_consensus.cc:729] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b439f9c3d53400e8e2dc7b5c0a724b4, State: Initialized, Role: LEARNER
I20240502 08:17:03.401978 25711 consensus_queue.cc:260] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: true } }
I20240502 08:17:03.402572 25711 ts_tablet_manager.cc:1398] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20240502 08:17:03.403535 23716 tablet_copy_service.cc:342] P 3de510fd6a8a47a6acc24acba19b2364: Request end of tablet copy session 4b439f9c3d53400e8e2dc7b5c0a724b4-eff43ec5b0ea4198ae23d57f2c4cb2c9 received from {username='slave'} at 127.20.207.6:37247
I20240502 08:17:03.403690 23716 tablet_copy_service.cc:433] P 3de510fd6a8a47a6acc24acba19b2364: ending tablet copy session 4b439f9c3d53400e8e2dc7b5c0a724b4-eff43ec5b0ea4198ae23d57f2c4cb2c9 on tablet eff43ec5b0ea4198ae23d57f2c4cb2c9 with peer 4b439f9c3d53400e8e2dc7b5c0a724b4
I20240502 08:17:03.409049 25757 tablet_bootstrap.cc:492] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:03.409636 25757 tablet_bootstrap.cc:492] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0: Bootstrap complete.
I20240502 08:17:03.409960 25757 ts_tablet_manager.cc:1367] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0: Time spent bootstrapping tablet: real 0.126s user 0.042s sys 0.008s
I20240502 08:17:03.411026 25757 raft_consensus.cc:348] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: true } }
I20240502 08:17:03.411345 25757 raft_consensus.cc:729] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 67e133ffa16d4a7a9ce63f934ab783b0, State: Initialized, Role: LEARNER
I20240502 08:17:03.411607 25757 consensus_queue.cc:260] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: true } }
I20240502 08:17:03.412210 25757 ts_tablet_manager.cc:1398] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:03.413259 23849 tablet_copy_service.cc:342] P dd7173a12d4a43b7bc728c0d99db5384: Request end of tablet copy session 67e133ffa16d4a7a9ce63f934ab783b0-c9d6f55ce8f547e395d686838dbb7bd2 received from {username='slave'} at 127.20.207.4:50257
I20240502 08:17:03.413427 23849 tablet_copy_service.cc:433] P dd7173a12d4a43b7bc728c0d99db5384: ending tablet copy session 67e133ffa16d4a7a9ce63f934ab783b0-c9d6f55ce8f547e395d686838dbb7bd2 on tablet c9d6f55ce8f547e395d686838dbb7bd2 with peer 67e133ffa16d4a7a9ce63f934ab783b0
I20240502 08:17:03.429783 25705 tablet_bootstrap.cc:492] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:03.430294 25705 tablet_bootstrap.cc:492] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0: Bootstrap complete.
I20240502 08:17:03.430572 25705 ts_tablet_manager.cc:1367] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0: Time spent bootstrapping tablet: real 0.097s user 0.045s sys 0.007s
I20240502 08:17:03.431600 25705 raft_consensus.cc:348] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: true } }
I20240502 08:17:03.431838 25705 raft_consensus.cc:729] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 962f20ef8dea4cac91c4bf7603a655c0, State: Initialized, Role: LEARNER
I20240502 08:17:03.432036 25705 consensus_queue.cc:260] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: true } }
I20240502 08:17:03.432586 25705 ts_tablet_manager.cc:1398] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:03.433486 23716 tablet_copy_service.cc:342] P 3de510fd6a8a47a6acc24acba19b2364: Request end of tablet copy session 962f20ef8dea4cac91c4bf7603a655c0-8096d61335f8493594e55720d47489bd received from {username='slave'} at 127.20.207.5:49045
I20240502 08:17:03.433624 23716 tablet_copy_service.cc:433] P 3de510fd6a8a47a6acc24acba19b2364: ending tablet copy session 962f20ef8dea4cac91c4bf7603a655c0-8096d61335f8493594e55720d47489bd on tablet 8096d61335f8493594e55720d47489bd with peer 962f20ef8dea4cac91c4bf7603a655c0
I20240502 08:17:03.447010 25445 raft_consensus.cc:1203] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:03.457930 25182 raft_consensus.cc:1203] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:03.524231 25052 raft_consensus.cc:1203] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:03.545190 24921 raft_consensus.cc:1203] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:03.619801 24921 raft_consensus.cc:1203] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:03.626209 25314 raft_consensus.cc:1203] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:03.630609 25052 raft_consensus.cc:1203] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:03.640333 25182 raft_consensus.cc:1203] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:03.678104 25445 raft_consensus.cc:1203] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:03.830240 25799 raft_consensus.cc:1050] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364: attempting to promote NON_VOTER cfb4a4db281241f5940203fe907dea50 to VOTER
I20240502 08:17:03.831090 25799 consensus_queue.cc:237] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } }
I20240502 08:17:03.833014 25445 raft_consensus.cc:1261] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 [term 1 LEARNER]: Refusing update from remote peer 3de510fd6a8a47a6acc24acba19b2364: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:03.833762 25802 consensus_queue.cc:1035] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:03.837435 25799 raft_consensus.cc:2932] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cfb4a4db281241f5940203fe907dea50 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } } }
I20240502 08:17:03.838229 25445 raft_consensus.cc:2932] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cfb4a4db281241f5940203fe907dea50 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } } }
I20240502 08:17:03.843233 23545 catalog_manager.cc:5484] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: config changed from index 4 to 5, cfb4a4db281241f5940203fe907dea50 (127.20.207.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:03.915303 25582 raft_consensus.cc:1050] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364: attempting to promote NON_VOTER 962f20ef8dea4cac91c4bf7603a655c0 to VOTER
I20240502 08:17:03.916080 25582 consensus_queue.cc:237] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } }
I20240502 08:17:03.918486 25182 raft_consensus.cc:1261] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 [term 1 LEARNER]: Refusing update from remote peer 3de510fd6a8a47a6acc24acba19b2364: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:03.919160 25802 consensus_queue.cc:1035] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Connected to new peer: Peer: permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:03.922962 25582 raft_consensus.cc:2932] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 962f20ef8dea4cac91c4bf7603a655c0 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } } }
I20240502 08:17:03.923873 25182 raft_consensus.cc:2932] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 962f20ef8dea4cac91c4bf7603a655c0 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } } }
I20240502 08:17:03.928761 23546 catalog_manager.cc:5484] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: config changed from index 4 to 5, 962f20ef8dea4cac91c4bf7603a655c0 (127.20.207.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:03.946141 25314 raft_consensus.cc:1203] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:17:04.001888 25582 raft_consensus.cc:1050] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364: attempting to promote NON_VOTER 67e133ffa16d4a7a9ce63f934ab783b0 to VOTER
I20240502 08:17:04.002562 25582 consensus_queue.cc:237] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } }
I20240502 08:17:04.004743 25052 raft_consensus.cc:1261] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 1 LEARNER]: Refusing update from remote peer 3de510fd6a8a47a6acc24acba19b2364: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:04.005367 25582 consensus_queue.cc:1035] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:04.009188 25799 raft_consensus.cc:2932] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 67e133ffa16d4a7a9ce63f934ab783b0 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } } }
I20240502 08:17:04.009972 25052 raft_consensus.cc:2932] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 67e133ffa16d4a7a9ce63f934ab783b0 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } } }
I20240502 08:17:04.013876 23546 catalog_manager.cc:5484] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: config changed from index 4 to 5, 67e133ffa16d4a7a9ce63f934ab783b0 (127.20.207.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.043536 25799 raft_consensus.cc:1050] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364: attempting to promote NON_VOTER 962f20ef8dea4cac91c4bf7603a655c0 to VOTER
I20240502 08:17:04.044173 25799 consensus_queue.cc:237] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } }
I20240502 08:17:04.046049 25182 raft_consensus.cc:1261] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 [term 1 LEARNER]: Refusing update from remote peer 3de510fd6a8a47a6acc24acba19b2364: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:04.046682 25582 consensus_queue.cc:1035] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Connected to new peer: Peer: permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:04.049441 25799 raft_consensus.cc:2932] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 962f20ef8dea4cac91c4bf7603a655c0 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } } }
I20240502 08:17:04.052950 25182 raft_consensus.cc:2932] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 962f20ef8dea4cac91c4bf7603a655c0 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } } }
I20240502 08:17:04.054349 23545 catalog_manager.cc:5484] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: config changed from index 4 to 5, 962f20ef8dea4cac91c4bf7603a655c0 (127.20.207.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.068534 23696 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c26c8082287402daf70ee9be5a4fb7f"
dest_uuid: "3de510fd6a8a47a6acc24acba19b2364"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54306
I20240502 08:17:04.068819 23696 raft_consensus.cc:595] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:04.079046 25820 raft_consensus.cc:1050] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384: attempting to promote NON_VOTER 67e133ffa16d4a7a9ce63f934ab783b0 to VOTER
I20240502 08:17:04.080094 25820 consensus_queue.cc:237] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } }
I20240502 08:17:04.082264 25052 raft_consensus.cc:1261] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 1 LEARNER]: Refusing update from remote peer dd7173a12d4a43b7bc728c0d99db5384: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:04.083078 25627 consensus_queue.cc:1035] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:04.086117 25820 raft_consensus.cc:2932] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 67e133ffa16d4a7a9ce63f934ab783b0 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } } }
I20240502 08:17:04.087843 23696 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7df5ab24ba3c42769dd4640220641996"
dest_uuid: "3de510fd6a8a47a6acc24acba19b2364"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54328
I20240502 08:17:04.088039 23696 raft_consensus.cc:595] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:04.090252 25052 raft_consensus.cc:2932] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 67e133ffa16d4a7a9ce63f934ab783b0 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } } }
I20240502 08:17:04.091280 23545 catalog_manager.cc:5484] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: config changed from index 4 to 5, 67e133ffa16d4a7a9ce63f934ab783b0 (127.20.207.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.101105 25820 raft_consensus.cc:1050] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384: attempting to promote NON_VOTER cfb4a4db281241f5940203fe907dea50 to VOTER
I20240502 08:17:04.101761 25820 consensus_queue.cc:237] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } }
I20240502 08:17:04.103796 25445 raft_consensus.cc:1261] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 [term 1 LEARNER]: Refusing update from remote peer dd7173a12d4a43b7bc728c0d99db5384: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:04.104377 25822 consensus_queue.cc:1035] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:04.107009 25820 raft_consensus.cc:2932] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cfb4a4db281241f5940203fe907dea50 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } } }
I20240502 08:17:04.109963 25445 raft_consensus.cc:2932] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cfb4a4db281241f5940203fe907dea50 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } } }
I20240502 08:17:04.110791 23696 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db6049ba7a5640d4ac4dae656b2f3667"
dest_uuid: "3de510fd6a8a47a6acc24acba19b2364"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54356
I20240502 08:17:04.111016 23696 raft_consensus.cc:595] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:04.111475 23546 catalog_manager.cc:5484] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: config changed from index 4 to 5, cfb4a4db281241f5940203fe907dea50 (127.20.207.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.130867 23696 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8096d61335f8493594e55720d47489bd"
dest_uuid: "3de510fd6a8a47a6acc24acba19b2364"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54374
I20240502 08:17:04.131105 23696 raft_consensus.cc:595] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:04.144719 25822 raft_consensus.cc:1050] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384: attempting to promote NON_VOTER 4b439f9c3d53400e8e2dc7b5c0a724b4 to VOTER
I20240502 08:17:04.145316 25822 consensus_queue.cc:237] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } }
I20240502 08:17:04.147311 25314 raft_consensus.cc:1261] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 1 LEARNER]: Refusing update from remote peer dd7173a12d4a43b7bc728c0d99db5384: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:04.147933 25627 consensus_queue.cc:1035] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:04.149117 25627 raft_consensus.cc:1050] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384: attempting to promote NON_VOTER 39c4a2d0ccc94109b6bd2cac1792003f to VOTER
I20240502 08:17:04.149747 25627 consensus_queue.cc:237] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } }
I20240502 08:17:04.151743 24921 raft_consensus.cc:1261] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f [term 1 LEARNER]: Refusing update from remote peer dd7173a12d4a43b7bc728c0d99db5384: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:04.151921 25627 raft_consensus.cc:2932] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b439f9c3d53400e8e2dc7b5c0a724b4 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } } }
I20240502 08:17:04.152335 25822 consensus_queue.cc:1035] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:04.152683 25314 raft_consensus.cc:2932] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4b439f9c3d53400e8e2dc7b5c0a724b4 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } } }
I20240502 08:17:04.157572 23545 catalog_manager.cc:5484] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: config changed from index 4 to 5, 4b439f9c3d53400e8e2dc7b5c0a724b4 (127.20.207.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.157586 25822 raft_consensus.cc:2932] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 39c4a2d0ccc94109b6bd2cac1792003f (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } } }
I20240502 08:17:04.159299 24921 raft_consensus.cc:2932] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 39c4a2d0ccc94109b6bd2cac1792003f (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } } }
I20240502 08:17:04.162760 23546 catalog_manager.cc:5484] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: config changed from index 4 to 5, 39c4a2d0ccc94109b6bd2cac1792003f (127.20.207.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.179970 25627 raft_consensus.cc:1050] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384: attempting to promote NON_VOTER 39c4a2d0ccc94109b6bd2cac1792003f to VOTER
I20240502 08:17:04.180550 25627 consensus_queue.cc:237] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } }
I20240502 08:17:04.182632 24921 raft_consensus.cc:1261] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f [term 1 LEARNER]: Refusing update from remote peer dd7173a12d4a43b7bc728c0d99db5384: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:04.183415 25820 consensus_queue.cc:1035] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:04.186865 25627 raft_consensus.cc:2932] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 39c4a2d0ccc94109b6bd2cac1792003f (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } } }
I20240502 08:17:04.189862 24921 raft_consensus.cc:2932] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 39c4a2d0ccc94109b6bd2cac1792003f (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } } }
I20240502 08:17:04.191538 23545 catalog_manager.cc:5484] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: config changed from index 4 to 5, 39c4a2d0ccc94109b6bd2cac1792003f (127.20.207.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.229133 25799 raft_consensus.cc:979] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364: : Instructing follower cfb4a4db281241f5940203fe907dea50 to start an election
I20240502 08:17:04.229372 25799 raft_consensus.cc:1067] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Signalling peer cfb4a4db281241f5940203fe907dea50 to start an election
I20240502 08:17:04.230327 25445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c26c8082287402daf70ee9be5a4fb7f"
dest_uuid: "cfb4a4db281241f5940203fe907dea50"
from {username='slave'} at 127.20.207.1:38983
I20240502 08:17:04.230643 25445 raft_consensus.cc:482] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:04.230813 25445 raft_consensus.cc:3037] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.232849 25445 raft_consensus.cc:504] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } }
I20240502 08:17:04.234107 25445 leader_election.cc:288] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 [CANDIDATE]: Term 2 election: Requested vote from peers 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519)
I20240502 08:17:04.234902 23696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c26c8082287402daf70ee9be5a4fb7f" candidate_uuid: "cfb4a4db281241f5940203fe907dea50" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3de510fd6a8a47a6acc24acba19b2364"
I20240502 08:17:04.235183 23696 raft_consensus.cc:3032] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:04.235289 23696 raft_consensus.cc:729] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Running, Role: LEADER
I20240502 08:17:04.235575 23696 consensus_queue.cc:260] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } }
I20240502 08:17:04.235872 23696 raft_consensus.cc:3037] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.237883 23696 raft_consensus.cc:2454] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfb4a4db281241f5940203fe907dea50 in term 2.
I20240502 08:17:04.238530 25378 leader_election.cc:302] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3de510fd6a8a47a6acc24acba19b2364, cfb4a4db281241f5940203fe907dea50; no voters:
I20240502 08:17:04.238940 25788 raft_consensus.cc:2781] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:04.239357 25788 raft_consensus.cc:686] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 [term 2 LEADER]: Becoming Leader. State: Replica: cfb4a4db281241f5940203fe907dea50, State: Running, Role: LEADER
I20240502 08:17:04.239806 25788 consensus_queue.cc:237] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } }
I20240502 08:17:04.243108 23546 catalog_manager.cc:5484] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 reported cstate change: term changed from 1 to 2, leader changed from 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1) to cfb4a4db281241f5940203fe907dea50 (127.20.207.7). New cstate: current_term: 2 leader_uuid: "cfb4a4db281241f5940203fe907dea50" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 08:17:04.254086 23540 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 1c26c8082287402daf70ee9be5a4fb7f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader cfb4a4db281241f5940203fe907dea50 (127.20.207.7:39999) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 08:17:04.367569 23828 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9d6f55ce8f547e395d686838dbb7bd2"
dest_uuid: "dd7173a12d4a43b7bc728c0d99db5384"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37618
I20240502 08:17:04.367851 23828 raft_consensus.cc:595] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:04.375911 25582 raft_consensus.cc:1050] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364: attempting to promote NON_VOTER 4b439f9c3d53400e8e2dc7b5c0a724b4 to VOTER
I20240502 08:17:04.376586 25582 consensus_queue.cc:237] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } }
I20240502 08:17:04.378602 25314 raft_consensus.cc:1261] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 1 LEARNER]: Refusing update from remote peer 3de510fd6a8a47a6acc24acba19b2364: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:04.379179 25799 consensus_queue.cc:1035] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:17:04.382143 25582 raft_consensus.cc:2932] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4b439f9c3d53400e8e2dc7b5c0a724b4 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } } }
I20240502 08:17:04.386193 25314 raft_consensus.cc:2932] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4b439f9c3d53400e8e2dc7b5c0a724b4 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } } }
I20240502 08:17:04.387122 23828 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb067e2ff17740958059cfef1e0d0f9d"
dest_uuid: "dd7173a12d4a43b7bc728c0d99db5384"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37638
I20240502 08:17:04.387331 23828 raft_consensus.cc:595] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:04.388015 23546 catalog_manager.cc:5484] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: config changed from index 2 to 3, 4b439f9c3d53400e8e2dc7b5c0a724b4 (127.20.207.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.414659 23828 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2c0157c3916c4012a6b8c4d38df7a342"
dest_uuid: "dd7173a12d4a43b7bc728c0d99db5384"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37670
I20240502 08:17:04.414897 23828 raft_consensus.cc:595] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:04.431387 25802 raft_consensus.cc:979] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364: : Instructing follower 67e133ffa16d4a7a9ce63f934ab783b0 to start an election
I20240502 08:17:04.431993 25582 raft_consensus.cc:1067] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Signalling peer 67e133ffa16d4a7a9ce63f934ab783b0 to start an election
I20240502 08:17:04.432822 25052 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7df5ab24ba3c42769dd4640220641996"
dest_uuid: "67e133ffa16d4a7a9ce63f934ab783b0"
from {username='slave'} at 127.20.207.1:41015
I20240502 08:17:04.433132 25052 raft_consensus.cc:482] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:04.433372 25052 raft_consensus.cc:3037] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.433490 23828 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ce947f6507ea457da00437f718444e5b"
dest_uuid: "dd7173a12d4a43b7bc728c0d99db5384"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37688
I20240502 08:17:04.433710 23828 raft_consensus.cc:595] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:04.437108 25052 raft_consensus.cc:504] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } }
I20240502 08:17:04.438441 25052 leader_election.cc:288] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 [CANDIDATE]: Term 2 election: Requested vote from peers 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519)
I20240502 08:17:04.439042 23696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df5ab24ba3c42769dd4640220641996" candidate_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3de510fd6a8a47a6acc24acba19b2364"
I20240502 08:17:04.439283 23696 raft_consensus.cc:3032] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:04.439379 23696 raft_consensus.cc:729] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Running, Role: LEADER
I20240502 08:17:04.439694 23696 consensus_queue.cc:260] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } }
I20240502 08:17:04.440009 23696 raft_consensus.cc:3037] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.442939 23696 raft_consensus.cc:2454] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e133ffa16d4a7a9ce63f934ab783b0 in term 2.
I20240502 08:17:04.443615 24986 leader_election.cc:302] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3de510fd6a8a47a6acc24acba19b2364, 67e133ffa16d4a7a9ce63f934ab783b0; no voters:
I20240502 08:17:04.444033 25790 raft_consensus.cc:2781] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:04.444463 25790 raft_consensus.cc:686] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 2 LEADER]: Becoming Leader. State: Replica: 67e133ffa16d4a7a9ce63f934ab783b0, State: Running, Role: LEADER
I20240502 08:17:04.444887 25790 consensus_queue.cc:237] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } }
I20240502 08:17:04.448026 23545 catalog_manager.cc:5484] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 reported cstate change: term changed from 1 to 2, leader changed from 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1) to 67e133ffa16d4a7a9ce63f934ab783b0 (127.20.207.4). New cstate: current_term: 2 leader_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.458321 25802 raft_consensus.cc:979] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364: : Instructing follower 962f20ef8dea4cac91c4bf7603a655c0 to start an election
I20240502 08:17:04.458524 25802 raft_consensus.cc:1067] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Signalling peer 962f20ef8dea4cac91c4bf7603a655c0 to start an election
I20240502 08:17:04.459925 25182 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8096d61335f8493594e55720d47489bd"
dest_uuid: "962f20ef8dea4cac91c4bf7603a655c0"
from {username='slave'} at 127.20.207.1:35919
I20240502 08:17:04.460253 25182 raft_consensus.cc:482] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:04.460462 25182 raft_consensus.cc:3037] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.460906 25582 raft_consensus.cc:979] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364: : Instructing follower 962f20ef8dea4cac91c4bf7603a655c0 to start an election
I20240502 08:17:04.461058 25582 raft_consensus.cc:1067] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Signalling peer 962f20ef8dea4cac91c4bf7603a655c0 to start an election
I20240502 08:17:04.461838 25181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db6049ba7a5640d4ac4dae656b2f3667"
dest_uuid: "962f20ef8dea4cac91c4bf7603a655c0"
from {username='slave'} at 127.20.207.1:35919
I20240502 08:17:04.462085 25181 raft_consensus.cc:482] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:04.462229 25181 raft_consensus.cc:3037] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.463209 25182 raft_consensus.cc:504] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } }
I20240502 08:17:04.464511 25182 leader_election.cc:288] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 [CANDIDATE]: Term 2 election: Requested vote from peers 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519)
I20240502 08:17:04.464937 23695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8096d61335f8493594e55720d47489bd" candidate_uuid: "962f20ef8dea4cac91c4bf7603a655c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3de510fd6a8a47a6acc24acba19b2364"
I20240502 08:17:04.465219 23695 raft_consensus.cc:3032] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:04.465166 25181 raft_consensus.cc:504] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } }
I20240502 08:17:04.465344 23695 raft_consensus.cc:729] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Running, Role: LEADER
I20240502 08:17:04.465660 25181 leader_election.cc:288] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 [CANDIDATE]: Term 2 election: Requested vote from peers 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519)
I20240502 08:17:04.465616 23695 consensus_queue.cc:260] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } }
I20240502 08:17:04.465924 23695 raft_consensus.cc:3037] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.466130 23696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db6049ba7a5640d4ac4dae656b2f3667" candidate_uuid: "962f20ef8dea4cac91c4bf7603a655c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3de510fd6a8a47a6acc24acba19b2364"
I20240502 08:17:04.466341 23696 raft_consensus.cc:3032] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:04.466432 23696 raft_consensus.cc:729] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Running, Role: LEADER
I20240502 08:17:04.466691 23696 consensus_queue.cc:260] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } }
I20240502 08:17:04.467056 23696 raft_consensus.cc:3037] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.469074 23695 raft_consensus.cc:2454] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 962f20ef8dea4cac91c4bf7603a655c0 in term 2.
I20240502 08:17:04.469602 25116 leader_election.cc:302] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3de510fd6a8a47a6acc24acba19b2364, 962f20ef8dea4cac91c4bf7603a655c0; no voters:
I20240502 08:17:04.469702 23696 raft_consensus.cc:2454] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 962f20ef8dea4cac91c4bf7603a655c0 in term 2.
I20240502 08:17:04.470360 25749 raft_consensus.cc:2781] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:04.470949 25116 leader_election.cc:302] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3de510fd6a8a47a6acc24acba19b2364, 962f20ef8dea4cac91c4bf7603a655c0; no voters:
I20240502 08:17:04.471196 26028 raft_consensus.cc:2781] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:04.471410 25749 raft_consensus.cc:686] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 [term 2 LEADER]: Becoming Leader. State: Replica: 962f20ef8dea4cac91c4bf7603a655c0, State: Running, Role: LEADER
I20240502 08:17:04.471917 25749 consensus_queue.cc:237] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } }
I20240502 08:17:04.474227 26028 raft_consensus.cc:686] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 [term 2 LEADER]: Becoming Leader. State: Replica: 962f20ef8dea4cac91c4bf7603a655c0, State: Running, Role: LEADER
I20240502 08:17:04.474553 26028 consensus_queue.cc:237] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } }
I20240502 08:17:04.476794 23545 catalog_manager.cc:5484] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 reported cstate change: term changed from 1 to 2, leader changed from 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1) to 962f20ef8dea4cac91c4bf7603a655c0 (127.20.207.5). New cstate: current_term: 2 leader_uuid: "962f20ef8dea4cac91c4bf7603a655c0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.482610 23545 catalog_manager.cc:5484] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 reported cstate change: term changed from 1 to 2, leader changed from 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1) to 962f20ef8dea4cac91c4bf7603a655c0 (127.20.207.5). New cstate: current_term: 2 leader_uuid: "962f20ef8dea4cac91c4bf7603a655c0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.511080 23828 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f214664b085a4ac0a10484acbd525142"
dest_uuid: "dd7173a12d4a43b7bc728c0d99db5384"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37694
I20240502 08:17:04.511354 23828 raft_consensus.cc:595] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:04.596957 25820 raft_consensus.cc:979] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384: : Instructing follower cfb4a4db281241f5940203fe907dea50 to start an election
I20240502 08:17:04.597177 25820 raft_consensus.cc:1067] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Signalling peer cfb4a4db281241f5940203fe907dea50 to start an election
I20240502 08:17:04.598124 25445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2c0157c3916c4012a6b8c4d38df7a342"
dest_uuid: "cfb4a4db281241f5940203fe907dea50"
from {username='slave'} at 127.20.207.2:51813
I20240502 08:17:04.598373 25445 raft_consensus.cc:482] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:04.598489 25445 raft_consensus.cc:3037] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.600502 25445 raft_consensus.cc:504] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } }
I20240502 08:17:04.601155 25445 leader_election.cc:288] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 [CANDIDATE]: Term 2 election: Requested vote from peers dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263)
I20240502 08:17:04.601825 23828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c0157c3916c4012a6b8c4d38df7a342" candidate_uuid: "cfb4a4db281241f5940203fe907dea50" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "dd7173a12d4a43b7bc728c0d99db5384"
I20240502 08:17:04.602119 23828 raft_consensus.cc:3032] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:04.602212 23828 raft_consensus.cc:729] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Running, Role: LEADER
I20240502 08:17:04.602530 23828 consensus_queue.cc:260] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } }
I20240502 08:17:04.602849 23828 raft_consensus.cc:3037] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.604817 23828 raft_consensus.cc:2454] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfb4a4db281241f5940203fe907dea50 in term 2.
I20240502 08:17:04.605314 25380 leader_election.cc:302] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: cfb4a4db281241f5940203fe907dea50, dd7173a12d4a43b7bc728c0d99db5384; no voters:
I20240502 08:17:04.605618 25788 raft_consensus.cc:2781] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:04.605769 25788 raft_consensus.cc:686] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 [term 2 LEADER]: Becoming Leader. State: Replica: cfb4a4db281241f5940203fe907dea50, State: Running, Role: LEADER
I20240502 08:17:04.606067 25788 consensus_queue.cc:237] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } }
I20240502 08:17:04.608137 23545 catalog_manager.cc:5484] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 reported cstate change: term changed from 1 to 2, leader changed from dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2) to cfb4a4db281241f5940203fe907dea50 (127.20.207.7). New cstate: current_term: 2 leader_uuid: "cfb4a4db281241f5940203fe907dea50" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.633626 23696 raft_consensus.cc:1261] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Refusing update from remote peer cfb4a4db281241f5940203fe907dea50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:17:04.634529 25788 consensus_queue.cc:1035] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:04.644430 25627 raft_consensus.cc:979] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384: : Instructing follower 67e133ffa16d4a7a9ce63f934ab783b0 to start an election
I20240502 08:17:04.644626 25627 raft_consensus.cc:1067] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Signalling peer 67e133ffa16d4a7a9ce63f934ab783b0 to start an election
I20240502 08:17:04.645220 25052 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9d6f55ce8f547e395d686838dbb7bd2"
dest_uuid: "67e133ffa16d4a7a9ce63f934ab783b0"
from {username='slave'} at 127.20.207.2:55741
I20240502 08:17:04.645468 25052 raft_consensus.cc:482] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:04.645592 25052 raft_consensus.cc:3037] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.647480 25052 raft_consensus.cc:504] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } }
I20240502 08:17:04.648041 25052 leader_election.cc:288] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 [CANDIDATE]: Term 2 election: Requested vote from peers dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263)
I20240502 08:17:04.648669 23828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d6f55ce8f547e395d686838dbb7bd2" candidate_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "dd7173a12d4a43b7bc728c0d99db5384"
I20240502 08:17:04.648876 23828 raft_consensus.cc:3032] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:04.648949 23828 raft_consensus.cc:729] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Running, Role: LEADER
I20240502 08:17:04.649192 23828 consensus_queue.cc:260] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } }
I20240502 08:17:04.649497 23828 raft_consensus.cc:3037] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.651377 23828 raft_consensus.cc:2454] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e133ffa16d4a7a9ce63f934ab783b0 in term 2.
I20240502 08:17:04.651821 24988 leader_election.cc:302] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67e133ffa16d4a7a9ce63f934ab783b0, dd7173a12d4a43b7bc728c0d99db5384; no voters:
I20240502 08:17:04.652122 26021 raft_consensus.cc:2781] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:04.652251 26021 raft_consensus.cc:686] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 2 LEADER]: Becoming Leader. State: Replica: 67e133ffa16d4a7a9ce63f934ab783b0, State: Running, Role: LEADER
I20240502 08:17:04.652576 26021 consensus_queue.cc:237] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } }
I20240502 08:17:04.654832 23546 catalog_manager.cc:5484] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 reported cstate change: term changed from 1 to 2, leader changed from dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2) to 67e133ffa16d4a7a9ce63f934ab783b0 (127.20.207.4). New cstate: current_term: 2 leader_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.689926 25820 raft_consensus.cc:979] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384: : Instructing follower 4b439f9c3d53400e8e2dc7b5c0a724b4 to start an election
I20240502 08:17:04.690124 25820 raft_consensus.cc:1067] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Signalling peer 4b439f9c3d53400e8e2dc7b5c0a724b4 to start an election
I20240502 08:17:04.690896 25314 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f214664b085a4ac0a10484acbd525142"
dest_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4"
from {username='slave'} at 127.20.207.2:58229
I20240502 08:17:04.691161 25314 raft_consensus.cc:482] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:04.691354 25314 raft_consensus.cc:3037] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.693354 25314 raft_consensus.cc:504] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } }
I20240502 08:17:04.694785 25314 leader_election.cc:288] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [CANDIDATE]: Term 2 election: Requested vote from peers dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263)
I20240502 08:17:04.695358 23828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f214664b085a4ac0a10484acbd525142" candidate_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "dd7173a12d4a43b7bc728c0d99db5384"
I20240502 08:17:04.695542 23828 raft_consensus.cc:3032] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:04.695631 23828 raft_consensus.cc:729] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Running, Role: LEADER
I20240502 08:17:04.695909 23828 consensus_queue.cc:260] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } }
I20240502 08:17:04.696168 23828 raft_consensus.cc:3037] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.698058 23828 raft_consensus.cc:2454] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b439f9c3d53400e8e2dc7b5c0a724b4 in term 2.
I20240502 08:17:04.698710 25249 leader_election.cc:302] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4b439f9c3d53400e8e2dc7b5c0a724b4, dd7173a12d4a43b7bc728c0d99db5384; no voters:
I20240502 08:17:04.699081 25755 raft_consensus.cc:2781] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:04.699498 25755 raft_consensus.cc:686] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 2 LEADER]: Becoming Leader. State: Replica: 4b439f9c3d53400e8e2dc7b5c0a724b4, State: Running, Role: LEADER
I20240502 08:17:04.699980 25755 consensus_queue.cc:237] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } }
I20240502 08:17:04.703325 23546 catalog_manager.cc:5484] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 reported cstate change: term changed from 1 to 2, leader changed from dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2) to 4b439f9c3d53400e8e2dc7b5c0a724b4 (127.20.207.6). New cstate: current_term: 2 leader_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.729470 23696 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eff43ec5b0ea4198ae23d57f2c4cb2c9"
dest_uuid: "3de510fd6a8a47a6acc24acba19b2364"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54424
I20240502 08:17:04.729699 23696 raft_consensus.cc:595] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:04.736598 25627 raft_consensus.cc:979] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384: : Instructing follower 39c4a2d0ccc94109b6bd2cac1792003f to start an election
I20240502 08:17:04.736797 25627 raft_consensus.cc:1067] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Signalling peer 39c4a2d0ccc94109b6bd2cac1792003f to start an election
I20240502 08:17:04.737581 24921 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce947f6507ea457da00437f718444e5b"
dest_uuid: "39c4a2d0ccc94109b6bd2cac1792003f"
from {username='slave'} at 127.20.207.2:60149
I20240502 08:17:04.737864 24921 raft_consensus.cc:482] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:04.738058 24921 raft_consensus.cc:3037] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.741528 24921 raft_consensus.cc:504] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } }
I20240502 08:17:04.743800 24921 leader_election.cc:288] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f [CANDIDATE]: Term 2 election: Requested vote from peers dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263)
I20240502 08:17:04.743883 23828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce947f6507ea457da00437f718444e5b" candidate_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "dd7173a12d4a43b7bc728c0d99db5384"
I20240502 08:17:04.744133 23828 raft_consensus.cc:3032] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:04.744256 23828 raft_consensus.cc:729] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Running, Role: LEADER
I20240502 08:17:04.744556 23828 consensus_queue.cc:260] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } }
I20240502 08:17:04.744839 23828 raft_consensus.cc:3037] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.747298 23828 raft_consensus.cc:2454] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39c4a2d0ccc94109b6bd2cac1792003f in term 2.
I20240502 08:17:04.747834 24857 leader_election.cc:302] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 39c4a2d0ccc94109b6bd2cac1792003f, dd7173a12d4a43b7bc728c0d99db5384; no voters:
I20240502 08:17:04.748518 26077 raft_consensus.cc:2781] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:04.749060 26077 raft_consensus.cc:686] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f [term 2 LEADER]: Becoming Leader. State: Replica: 39c4a2d0ccc94109b6bd2cac1792003f, State: Running, Role: LEADER
I20240502 08:17:04.749629 26077 consensus_queue.cc:237] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } }
I20240502 08:17:04.750654 25627 raft_consensus.cc:979] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384: : Instructing follower 39c4a2d0ccc94109b6bd2cac1792003f to start an election
I20240502 08:17:04.750833 25627 raft_consensus.cc:1067] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Signalling peer 39c4a2d0ccc94109b6bd2cac1792003f to start an election
I20240502 08:17:04.751415 24921 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb067e2ff17740958059cfef1e0d0f9d"
dest_uuid: "39c4a2d0ccc94109b6bd2cac1792003f"
from {username='slave'} at 127.20.207.2:60149
I20240502 08:17:04.751597 24921 raft_consensus.cc:482] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:04.751710 24921 raft_consensus.cc:3037] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.754246 24921 raft_consensus.cc:504] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } }
I20240502 08:17:04.754696 24921 leader_election.cc:288] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f [CANDIDATE]: Term 2 election: Requested vote from peers dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263)
I20240502 08:17:04.755172 23828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb067e2ff17740958059cfef1e0d0f9d" candidate_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "dd7173a12d4a43b7bc728c0d99db5384"
I20240502 08:17:04.755389 23828 raft_consensus.cc:3032] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:04.755486 23828 raft_consensus.cc:729] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Running, Role: LEADER
I20240502 08:17:04.755766 23828 consensus_queue.cc:260] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } }
I20240502 08:17:04.756150 23828 raft_consensus.cc:3037] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.758141 23546 catalog_manager.cc:5484] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f reported cstate change: term changed from 1 to 2, leader changed from dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2) to 39c4a2d0ccc94109b6bd2cac1792003f (127.20.207.3). New cstate: current_term: 2 leader_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.759297 23828 raft_consensus.cc:2454] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39c4a2d0ccc94109b6bd2cac1792003f in term 2.
I20240502 08:17:04.760946 24857 leader_election.cc:302] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 39c4a2d0ccc94109b6bd2cac1792003f, dd7173a12d4a43b7bc728c0d99db5384; no voters:
I20240502 08:17:04.761513 26077 raft_consensus.cc:2781] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:04.761807 26077 raft_consensus.cc:686] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f [term 2 LEADER]: Becoming Leader. State: Replica: 39c4a2d0ccc94109b6bd2cac1792003f, State: Running, Role: LEADER
I20240502 08:17:04.762209 26077 consensus_queue.cc:237] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } }
I20240502 08:17:04.765107 23546 catalog_manager.cc:5484] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f reported cstate change: term changed from 1 to 2, leader changed from dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2) to 39c4a2d0ccc94109b6bd2cac1792003f (127.20.207.3). New cstate: current_term: 2 leader_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.771941 23828 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb067e2ff17740958059cfef1e0d0f9d"
dest_uuid: "dd7173a12d4a43b7bc728c0d99db5384"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37712
I20240502 08:17:04.772200 23828 raft_consensus.cc:595] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Received request to transfer leadership
I20240502 08:17:04.772305 23828 raft_consensus.cc:603] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240502 08:17:04.908126 25445 consensus_queue.cc:237] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } }
I20240502 08:17:04.908982 25966 raft_consensus.cc:2932] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } } }
I20240502 08:17:04.911733 23540 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c26c8082287402daf70ee9be5a4fb7f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 08:17:04.913015 23546 catalog_manager.cc:5484] T 1c26c8082287402daf70ee9be5a4fb7f P cfb4a4db281241f5940203fe907dea50 reported cstate change: config changed from index 5 to 7, VOTER 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1) evicted. New cstate: current_term: 2 leader_uuid: "cfb4a4db281241f5940203fe907dea50" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.919667 23675 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c26c8082287402daf70ee9be5a4fb7f with delete_type TABLET_DATA_TOMBSTONED (TS 3de510fd6a8a47a6acc24acba19b2364 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
I20240502 08:17:04.920172 25799 raft_consensus.cc:979] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364: : Instructing follower 4b439f9c3d53400e8e2dc7b5c0a724b4 to start an election
I20240502 08:17:04.920347 25799 raft_consensus.cc:1067] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Signalling peer 4b439f9c3d53400e8e2dc7b5c0a724b4 to start an election
I20240502 08:17:04.920984 26120 tablet_replica.cc:330] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364: stopping tablet replica
I20240502 08:17:04.921100 25314 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eff43ec5b0ea4198ae23d57f2c4cb2c9"
dest_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4"
from {username='slave'} at 127.20.207.1:35555
I20240502 08:17:04.921308 25314 raft_consensus.cc:482] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:04.921413 25314 raft_consensus.cc:3037] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.921433 26120 raft_consensus.cc:2229] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:04.921720 26120 raft_consensus.cc:2258] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.923223 26120 ts_tablet_manager.cc:1875] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:04.923554 25314 raft_consensus.cc:504] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } }
I20240502 08:17:04.924118 25314 leader_election.cc:288] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [CANDIDATE]: Term 2 election: Requested vote from peers 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519)
I20240502 08:17:04.924741 23696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff43ec5b0ea4198ae23d57f2c4cb2c9" candidate_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3de510fd6a8a47a6acc24acba19b2364"
I20240502 08:17:04.924964 23696 raft_consensus.cc:3032] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:04.925067 23696 raft_consensus.cc:729] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Running, Role: LEADER
I20240502 08:17:04.925318 23696 consensus_queue.cc:260] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } }
I20240502 08:17:04.925580 23696 raft_consensus.cc:3037] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:04.927732 23696 raft_consensus.cc:2454] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b439f9c3d53400e8e2dc7b5c0a724b4 in term 2.
I20240502 08:17:04.928194 25247 leader_election.cc:302] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3de510fd6a8a47a6acc24acba19b2364, 4b439f9c3d53400e8e2dc7b5c0a724b4; no voters:
I20240502 08:17:04.928534 25755 raft_consensus.cc:2781] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:04.928710 25755 raft_consensus.cc:686] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 2 LEADER]: Becoming Leader. State: Replica: 4b439f9c3d53400e8e2dc7b5c0a724b4, State: Running, Role: LEADER
I20240502 08:17:04.928913 26120 ts_tablet_manager.cc:1888] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:17:04.928932 25755 consensus_queue.cc:237] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } }
I20240502 08:17:04.929066 26120 log.cc:1196] T 1c26c8082287402daf70ee9be5a4fb7f P 3de510fd6a8a47a6acc24acba19b2364: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/1c26c8082287402daf70ee9be5a4fb7f
I20240502 08:17:04.930651 23538 catalog_manager.cc:4830] TS 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519): tablet 1c26c8082287402daf70ee9be5a4fb7f (table rebalance_test_table_3 [id=0a34792f1dc944fea6b6ce2a62a12693]) successfully deleted
I20240502 08:17:04.931656 23546 catalog_manager.cc:5484] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 reported cstate change: term changed from 1 to 2, leader changed from 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1) to 4b439f9c3d53400e8e2dc7b5c0a724b4 (127.20.207.6). New cstate: current_term: 2 leader_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.964598 23696 raft_consensus.cc:1261] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Refusing update from remote peer 962f20ef8dea4cac91c4bf7603a655c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:17:04.965299 26028 consensus_queue.cc:1035] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:05.036902 23828 raft_consensus.cc:1261] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Refusing update from remote peer 67e133ffa16d4a7a9ce63f934ab783b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:17:05.037843 26021 consensus_queue.cc:1035] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:05.047036 23696 raft_consensus.cc:1261] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Refusing update from remote peer 67e133ffa16d4a7a9ce63f934ab783b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:17:05.047598 25790 consensus_queue.cc:1035] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:05.076977 23696 raft_consensus.cc:1261] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Refusing update from remote peer 962f20ef8dea4cac91c4bf7603a655c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:17:05.077661 26033 consensus_queue.cc:1035] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:05.084091 23828 raft_consensus.cc:1261] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Refusing update from remote peer cfb4a4db281241f5940203fe907dea50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:17:05.084704 25966 consensus_queue.cc:1035] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:05.106684 25052 consensus_queue.cc:237] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } }
I20240502 08:17:05.107671 25790 raft_consensus.cc:2932] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } } }
I20240502 08:17:05.111011 23539 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7df5ab24ba3c42769dd4640220641996 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 08:17:05.112952 23546 catalog_manager.cc:5484] T 7df5ab24ba3c42769dd4640220641996 P 67e133ffa16d4a7a9ce63f934ab783b0 reported cstate change: config changed from index 5 to 7, VOTER 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1) evicted. New cstate: current_term: 2 leader_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:05.116690 23675 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df5ab24ba3c42769dd4640220641996 with delete_type TABLET_DATA_TOMBSTONED (TS 3de510fd6a8a47a6acc24acba19b2364 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
I20240502 08:17:05.117018 26120 tablet_replica.cc:330] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364: stopping tablet replica
I20240502 08:17:05.117197 26120 raft_consensus.cc:2229] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.117410 26120 raft_consensus.cc:2258] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.118479 26120 ts_tablet_manager.cc:1875] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:05.125722 26120 ts_tablet_manager.cc:1888] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:17:05.125905 26120 log.cc:1196] T 7df5ab24ba3c42769dd4640220641996 P 3de510fd6a8a47a6acc24acba19b2364: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/7df5ab24ba3c42769dd4640220641996
I20240502 08:17:05.127106 23538 catalog_manager.cc:4830] TS 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519): tablet 7df5ab24ba3c42769dd4640220641996 (table rebalance_test_table_1 [id=d6ca06ec7e9d49f88451cc03d5275d15]) successfully deleted
I20240502 08:17:05.128692 25182 consensus_queue.cc:237] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } }
I20240502 08:17:05.129695 26028 raft_consensus.cc:2932] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } } }
I20240502 08:17:05.133405 23538 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8096d61335f8493594e55720d47489bd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 08:17:05.134959 23546 catalog_manager.cc:5484] T 8096d61335f8493594e55720d47489bd P 962f20ef8dea4cac91c4bf7603a655c0 reported cstate change: config changed from index 5 to 7, VOTER 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1) evicted. New cstate: current_term: 2 leader_uuid: "962f20ef8dea4cac91c4bf7603a655c0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:05.138830 23675 tablet_service.cc:1514] Processing DeleteTablet for tablet 8096d61335f8493594e55720d47489bd with delete_type TABLET_DATA_TOMBSTONED (TS 3de510fd6a8a47a6acc24acba19b2364 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
I20240502 08:17:05.139206 26120 tablet_replica.cc:330] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364: stopping tablet replica
I20240502 08:17:05.139403 26120 raft_consensus.cc:2229] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.139596 26120 raft_consensus.cc:2258] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.140398 26120 ts_tablet_manager.cc:1875] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:05.145256 26120 ts_tablet_manager.cc:1888] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:17:05.145407 26120 log.cc:1196] T 8096d61335f8493594e55720d47489bd P 3de510fd6a8a47a6acc24acba19b2364: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/8096d61335f8493594e55720d47489bd
I20240502 08:17:05.146346 23538 catalog_manager.cc:4830] TS 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519): tablet 8096d61335f8493594e55720d47489bd (table rebalance_test_table_3 [id=0a34792f1dc944fea6b6ce2a62a12693]) successfully deleted
I20240502 08:17:05.155464 25182 consensus_queue.cc:237] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } }
I20240502 08:17:05.156672 26028 raft_consensus.cc:2932] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } } }
I20240502 08:17:05.159799 23538 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet db6049ba7a5640d4ac4dae656b2f3667 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 08:17:05.161195 23546 catalog_manager.cc:5484] T db6049ba7a5640d4ac4dae656b2f3667 P 962f20ef8dea4cac91c4bf7603a655c0 reported cstate change: config changed from index 5 to 7, VOTER 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1) evicted. New cstate: current_term: 2 leader_uuid: "962f20ef8dea4cac91c4bf7603a655c0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "962f20ef8dea4cac91c4bf7603a655c0" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 44257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:05.165719 23675 tablet_service.cc:1514] Processing DeleteTablet for tablet db6049ba7a5640d4ac4dae656b2f3667 with delete_type TABLET_DATA_TOMBSTONED (TS 3de510fd6a8a47a6acc24acba19b2364 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
I20240502 08:17:05.166242 26120 tablet_replica.cc:330] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364: stopping tablet replica
I20240502 08:17:05.166478 26120 raft_consensus.cc:2229] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.166699 26120 raft_consensus.cc:2258] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.167632 26120 ts_tablet_manager.cc:1875] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:05.171953 26120 ts_tablet_manager.cc:1888] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:17:05.172080 26120 log.cc:1196] T db6049ba7a5640d4ac4dae656b2f3667 P 3de510fd6a8a47a6acc24acba19b2364: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/db6049ba7a5640d4ac4dae656b2f3667
I20240502 08:17:05.172960 23538 catalog_manager.cc:4830] TS 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519): tablet db6049ba7a5640d4ac4dae656b2f3667 (table rebalance_test_table_1 [id=d6ca06ec7e9d49f88451cc03d5275d15]) successfully deleted
I20240502 08:17:05.181174 23828 raft_consensus.cc:1261] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Refusing update from remote peer 4b439f9c3d53400e8e2dc7b5c0a724b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:17:05.181912 26055 consensus_queue.cc:1035] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:05.218843 23828 raft_consensus.cc:1261] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Refusing update from remote peer 39c4a2d0ccc94109b6bd2cac1792003f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:17:05.219542 26077 consensus_queue.cc:1035] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:05.254642 25445 consensus_queue.cc:237] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } }
I20240502 08:17:05.255669 25788 raft_consensus.cc:2932] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } } }
I20240502 08:17:05.259248 23540 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2c0157c3916c4012a6b8c4d38df7a342 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 08:17:05.260111 25052 consensus_queue.cc:237] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } }
I20240502 08:17:05.260951 23545 catalog_manager.cc:5484] T 2c0157c3916c4012a6b8c4d38df7a342 P cfb4a4db281241f5940203fe907dea50 reported cstate change: config changed from index 5 to 7, VOTER dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2) evicted. New cstate: current_term: 2 leader_uuid: "cfb4a4db281241f5940203fe907dea50" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cfb4a4db281241f5940203fe907dea50" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 39999 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:05.260994 25790 raft_consensus.cc:2932] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } } }
I20240502 08:17:05.264737 23539 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c9d6f55ce8f547e395d686838dbb7bd2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 08:17:05.265784 23808 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c0157c3916c4012a6b8c4d38df7a342 with delete_type TABLET_DATA_TOMBSTONED (TS dd7173a12d4a43b7bc728c0d99db5384 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49554
I20240502 08:17:05.266999 23545 catalog_manager.cc:5484] T c9d6f55ce8f547e395d686838dbb7bd2 P 67e133ffa16d4a7a9ce63f934ab783b0 reported cstate change: config changed from index 5 to 7, VOTER dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2) evicted. New cstate: current_term: 2 leader_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "67e133ffa16d4a7a9ce63f934ab783b0" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 43335 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:05.271517 23807 tablet_service.cc:1514] Processing DeleteTablet for tablet c9d6f55ce8f547e395d686838dbb7bd2 with delete_type TABLET_DATA_TOMBSTONED (TS dd7173a12d4a43b7bc728c0d99db5384 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49554
I20240502 08:17:05.272383 26218 tablet_replica.cc:330] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384: stopping tablet replica
I20240502 08:17:05.272884 26218 raft_consensus.cc:2229] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.273340 26218 raft_consensus.cc:2258] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.275772 26218 ts_tablet_manager.cc:1875] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:05.282788 26218 ts_tablet_manager.cc:1888] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:17:05.283031 26218 log.cc:1196] T 2c0157c3916c4012a6b8c4d38df7a342 P dd7173a12d4a43b7bc728c0d99db5384: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/2c0157c3916c4012a6b8c4d38df7a342
I20240502 08:17:05.283962 26218 tablet_replica.cc:330] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384: stopping tablet replica
I20240502 08:17:05.284065 23540 catalog_manager.cc:4830] TS dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263): tablet 2c0157c3916c4012a6b8c4d38df7a342 (table rebalance_test_table_4 [id=ca0b1080aa984ac38b4aa6099730ddce]) successfully deleted
I20240502 08:17:05.284256 26218 raft_consensus.cc:2229] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.284456 26218 raft_consensus.cc:2258] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.285310 26218 ts_tablet_manager.cc:1875] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:05.290061 26218 ts_tablet_manager.cc:1888] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:17:05.290287 26218 log.cc:1196] T c9d6f55ce8f547e395d686838dbb7bd2 P dd7173a12d4a43b7bc728c0d99db5384: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/c9d6f55ce8f547e395d686838dbb7bd2
I20240502 08:17:05.291447 23540 catalog_manager.cc:4830] TS dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263): tablet c9d6f55ce8f547e395d686838dbb7bd2 (table rebalance_test_table_2 [id=caae7f99bbcf4a949362006ef7059b72]) successfully deleted
W20240502 08:17:05.313103 23539 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet eff43ec5b0ea4198ae23d57f2c4cb2c9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 4b439f9c3d53400e8e2dc7b5c0a724b4 (127.20.207.6:42387) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 08:17:05.344214 25314 consensus_queue.cc:237] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } }
I20240502 08:17:05.345217 26055 raft_consensus.cc:2932] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } } }
I20240502 08:17:05.348270 23539 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f214664b085a4ac0a10484acbd525142 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 08:17:05.349918 23545 catalog_manager.cc:5484] T f214664b085a4ac0a10484acbd525142 P 4b439f9c3d53400e8e2dc7b5c0a724b4 reported cstate change: config changed from index 5 to 7, VOTER dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2) evicted. New cstate: current_term: 2 leader_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:05.354266 23808 tablet_service.cc:1514] Processing DeleteTablet for tablet f214664b085a4ac0a10484acbd525142 with delete_type TABLET_DATA_TOMBSTONED (TS dd7173a12d4a43b7bc728c0d99db5384 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49554
I20240502 08:17:05.354692 26218 tablet_replica.cc:330] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384: stopping tablet replica
I20240502 08:17:05.354892 26218 raft_consensus.cc:2229] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.355101 26218 raft_consensus.cc:2258] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.356127 26218 ts_tablet_manager.cc:1875] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:05.361683 26218 ts_tablet_manager.cc:1888] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:17:05.361825 26218 log.cc:1196] T f214664b085a4ac0a10484acbd525142 P dd7173a12d4a43b7bc728c0d99db5384: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/f214664b085a4ac0a10484acbd525142
I20240502 08:17:05.362646 23540 catalog_manager.cc:4830] TS dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263): tablet f214664b085a4ac0a10484acbd525142 (table rebalance_test_table_0 [id=c281f5d4d5124b60befcabf1e97b35a9]) successfully deleted
I20240502 08:17:05.371547 24920 consensus_queue.cc:237] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } }
I20240502 08:17:05.372361 23828 raft_consensus.cc:1261] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Refusing update from remote peer 39c4a2d0ccc94109b6bd2cac1792003f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:17:05.372668 26078 raft_consensus.cc:2932] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } } }
I20240502 08:17:05.373041 26077 consensus_queue.cc:1035] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:05.376886 23538 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fb067e2ff17740958059cfef1e0d0f9d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 08:17:05.377647 23544 catalog_manager.cc:5484] T fb067e2ff17740958059cfef1e0d0f9d P 39c4a2d0ccc94109b6bd2cac1792003f reported cstate change: config changed from index 5 to 7, VOTER dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2) evicted. New cstate: current_term: 2 leader_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:05.382053 23808 tablet_service.cc:1514] Processing DeleteTablet for tablet fb067e2ff17740958059cfef1e0d0f9d with delete_type TABLET_DATA_TOMBSTONED (TS dd7173a12d4a43b7bc728c0d99db5384 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49554
I20240502 08:17:05.382409 26218 tablet_replica.cc:330] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384: stopping tablet replica
I20240502 08:17:05.382604 26218 raft_consensus.cc:2229] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.382838 26218 raft_consensus.cc:2258] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.384459 26218 ts_tablet_manager.cc:1875] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:05.389299 26218 ts_tablet_manager.cc:1888] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:17:05.389426 26218 log.cc:1196] T fb067e2ff17740958059cfef1e0d0f9d P dd7173a12d4a43b7bc728c0d99db5384: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/fb067e2ff17740958059cfef1e0d0f9d
I20240502 08:17:05.390148 23540 catalog_manager.cc:4830] TS dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263): tablet fb067e2ff17740958059cfef1e0d0f9d (table rebalance_test_table_2 [id=caae7f99bbcf4a949362006ef7059b72]) successfully deleted
I20240502 08:17:05.531666 23696 raft_consensus.cc:1261] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Refusing update from remote peer 4b439f9c3d53400e8e2dc7b5c0a724b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:17:05.532245 26187 consensus_queue.cc:1035] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:05.583977 25314 consensus_queue.cc:237] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } }
I20240502 08:17:05.584900 26055 raft_consensus.cc:2932] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } } }
I20240502 08:17:05.587913 23539 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eff43ec5b0ea4198ae23d57f2c4cb2c9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 08:17:05.589150 23545 catalog_manager.cc:5484] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 4b439f9c3d53400e8e2dc7b5c0a724b4 reported cstate change: config changed from index 3 to 5, VOTER 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1) evicted. New cstate: current_term: 2 leader_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b439f9c3d53400e8e2dc7b5c0a724b4" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 42387 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:05.592473 23675 tablet_service.cc:1514] Processing DeleteTablet for tablet eff43ec5b0ea4198ae23d57f2c4cb2c9 with delete_type TABLET_DATA_TOMBSTONED (TS 3de510fd6a8a47a6acc24acba19b2364 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60256
I20240502 08:17:05.592797 26120 tablet_replica.cc:330] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364: stopping tablet replica
I20240502 08:17:05.592953 26120 raft_consensus.cc:2229] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.593109 26120 raft_consensus.cc:2258] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.593760 26120 ts_tablet_manager.cc:1875] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:05.598194 26120 ts_tablet_manager.cc:1888] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 08:17:05.598313 26120 log.cc:1196] T eff43ec5b0ea4198ae23d57f2c4cb2c9 P 3de510fd6a8a47a6acc24acba19b2364: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/eff43ec5b0ea4198ae23d57f2c4cb2c9
I20240502 08:17:05.599050 23538 catalog_manager.cc:4830] TS 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519): tablet eff43ec5b0ea4198ae23d57f2c4cb2c9 (table TestTable [id=3b4c053dcc94451bae4c42fa985e8559]) successfully deleted
I20240502 08:17:05.915123 24920 consensus_queue.cc:237] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } }
I20240502 08:17:05.916826 26319 raft_consensus.cc:2932] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } } }
I20240502 08:17:05.919965 23538 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ce947f6507ea457da00437f718444e5b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 7)
I20240502 08:17:05.921670 23545 catalog_manager.cc:5484] T ce947f6507ea457da00437f718444e5b P 39c4a2d0ccc94109b6bd2cac1792003f reported cstate change: config changed from index 5 to 7, VOTER dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2) evicted. New cstate: current_term: 2 leader_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39c4a2d0ccc94109b6bd2cac1792003f" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 45649 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:05.925298 23808 tablet_service.cc:1514] Processing DeleteTablet for tablet ce947f6507ea457da00437f718444e5b with delete_type TABLET_DATA_TOMBSTONED (TS dd7173a12d4a43b7bc728c0d99db5384 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49554
I20240502 08:17:05.925899 26321 tablet_replica.cc:330] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384: stopping tablet replica
I20240502 08:17:05.926116 26321 raft_consensus.cc:2229] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.926322 26321 raft_consensus.cc:2258] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.927048 26321 ts_tablet_manager.cc:1875] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:05.931525 26321 ts_tablet_manager.cc:1888] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:17:05.931633 26321 log.cc:1196] T ce947f6507ea457da00437f718444e5b P dd7173a12d4a43b7bc728c0d99db5384: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/ce947f6507ea457da00437f718444e5b
I20240502 08:17:05.932404 23540 catalog_manager.cc:4830] TS dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263): tablet ce947f6507ea457da00437f718444e5b (table rebalance_test_table_4 [id=ca0b1080aa984ac38b4aa6099730ddce]) successfully deleted
I20240502 08:17:06.205911 24901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:06.223207 23675 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:06.241035 25162 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:06.243120 25032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:06.264444 25293 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:06.305804 23808 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:06.306721 25425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:06.453187 23675 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:06.462638 25293 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:06.462983 24901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:06.464790 25032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:06.467237 25425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:06.480901 23808 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:06.510777 25162 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:07.156327 24901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:07.160604 23675 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:07.189286 25293 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:07.197204 25032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:07.198956 25162 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:07.210233 23808 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:07.233896 25425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:07.431561 24901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:07.431854 25293 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:07.449141 23675 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:07.450829 25032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:07.472234 25162 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:07.491889 25425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:07.511549 23808 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:07.761524 23675 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:07.763806 24901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:07.764556 25293 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:07.779762 25425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:07.783681 25162 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:07.786121 25032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:07.788079 23808 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
7eb46d820d5744549fe812ddd6fca0b0 | 127.20.207.62:36471 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.20.207.20:37025 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
39c4a2d0ccc94109b6bd2cac1792003f | 127.20.207.3:45649 | HEALTHY | <none> | 2 | 0
3de510fd6a8a47a6acc24acba19b2364 | 127.20.207.1:41519 | HEALTHY | <none> | 3 | 0
4b439f9c3d53400e8e2dc7b5c0a724b4 | 127.20.207.6:42387 | HEALTHY | <none> | 2 | 0
67e133ffa16d4a7a9ce63f934ab783b0 | 127.20.207.4:43335 | HEALTHY | <none> | 2 | 0
962f20ef8dea4cac91c4bf7603a655c0 | 127.20.207.5:44257 | HEALTHY | <none> | 2 | 0
cfb4a4db281241f5940203fe907dea50 | 127.20.207.7:39999 | HEALTHY | <none> | 2 | 0
dd7173a12d4a43b7bc728c0d99db5384 | 127.20.207.2:35263 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.20.207.1 | experimental | 127.20.207.1:41519
local_ip_for_outbound_sockets | 127.20.207.2 | experimental | 127.20.207.2:35263
local_ip_for_outbound_sockets | 127.20.207.3 | experimental | 127.20.207.3:45649
local_ip_for_outbound_sockets | 127.20.207.4 | experimental | 127.20.207.4:43335
local_ip_for_outbound_sockets | 127.20.207.5 | experimental | 127.20.207.5:44257
local_ip_for_outbound_sockets | 127.20.207.6 | experimental | 127.20.207.6:42387
local_ip_for_outbound_sockets | 127.20.207.7 | experimental | 127.20.207.7:39999
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.20.207.1:41519
server_dump_info_path | /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.20.207.2:35263
server_dump_info_path | /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.20.207.3:45649
server_dump_info_path | /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.20.207.4:43335
server_dump_info_path | /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.20.207.5:44257
server_dump_info_path | /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.20.207.6:42387
server_dump_info_path | /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.20.207.7:39999
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.20.207.20:37025 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 08:17:07.873893 21308 log_verifier.cc:126] Checking tablet 1c26c8082287402daf70ee9be5a4fb7f
I20240502 08:17:07.889447 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 1c26c8082287402daf70ee9be5a4fb7f
I20240502 08:17:07.889631 21308 log_verifier.cc:126] Checking tablet 2c0157c3916c4012a6b8c4d38df7a342
I20240502 08:17:07.901665 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 2c0157c3916c4012a6b8c4d38df7a342
I20240502 08:17:07.901778 21308 log_verifier.cc:126] Checking tablet 7df5ab24ba3c42769dd4640220641996
I20240502 08:17:07.919129 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 7df5ab24ba3c42769dd4640220641996
I20240502 08:17:07.919253 21308 log_verifier.cc:126] Checking tablet 8096d61335f8493594e55720d47489bd
I20240502 08:17:07.933579 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 8096d61335f8493594e55720d47489bd
I20240502 08:17:07.933692 21308 log_verifier.cc:126] Checking tablet 9506dfea92e3488fbe504216a75c4798
I20240502 08:17:07.945617 21308 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9506dfea92e3488fbe504216a75c4798
I20240502 08:17:07.945721 21308 log_verifier.cc:126] Checking tablet 9a4dfe88384b446fa08524fa91e557b7
I20240502 08:17:07.958128 21308 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9a4dfe88384b446fa08524fa91e557b7
I20240502 08:17:07.958225 21308 log_verifier.cc:126] Checking tablet ac96d624768a4d26bc2cbc43d120d3d1
I20240502 08:17:07.969684 21308 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ac96d624768a4d26bc2cbc43d120d3d1
I20240502 08:17:07.969786 21308 log_verifier.cc:126] Checking tablet b15e8c74b3b448538e315598c323601a
I20240502 08:17:07.981576 21308 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b15e8c74b3b448538e315598c323601a
I20240502 08:17:07.981675 21308 log_verifier.cc:126] Checking tablet bf84e851bd9e41acb2fc657fb516cb6e
I20240502 08:17:07.993541 21308 log_verifier.cc:177] Verified matching terms for 3 ops in tablet bf84e851bd9e41acb2fc657fb516cb6e
I20240502 08:17:07.993672 21308 log_verifier.cc:126] Checking tablet c9d6f55ce8f547e395d686838dbb7bd2
I20240502 08:17:08.005362 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c9d6f55ce8f547e395d686838dbb7bd2
I20240502 08:17:08.005465 21308 log_verifier.cc:126] Checking tablet ce947f6507ea457da00437f718444e5b
I20240502 08:17:08.017416 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet ce947f6507ea457da00437f718444e5b
I20240502 08:17:08.017531 21308 log_verifier.cc:126] Checking tablet db6049ba7a5640d4ac4dae656b2f3667
I20240502 08:17:08.030730 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet db6049ba7a5640d4ac4dae656b2f3667
I20240502 08:17:08.030869 21308 log_verifier.cc:126] Checking tablet eff43ec5b0ea4198ae23d57f2c4cb2c9
I20240502 08:17:08.039151 21308 log_verifier.cc:177] Verified matching terms for 5 ops in tablet eff43ec5b0ea4198ae23d57f2c4cb2c9
I20240502 08:17:08.039264 21308 log_verifier.cc:126] Checking tablet f214664b085a4ac0a10484acbd525142
I20240502 08:17:08.051496 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet f214664b085a4ac0a10484acbd525142
I20240502 08:17:08.051599 21308 log_verifier.cc:126] Checking tablet f6dd3894b6aa489f92fcb1aa383b2a58
I20240502 08:17:08.063876 21308 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f6dd3894b6aa489f92fcb1aa383b2a58
I20240502 08:17:08.063987 21308 log_verifier.cc:126] Checking tablet fb067e2ff17740958059cfef1e0d0f9d
I20240502 08:17:08.076385 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet fb067e2ff17740958059cfef1e0d0f9d
I20240502 08:17:08.079382 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.8:0
--local_ip_for_outbound_sockets=127.20.207.8
--webserver_interface=127.20.207.8
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36471
--builtin_ntp_servers=127.20.207.20:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:17:08.419653 26512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:08.419967 26512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:08.420120 26512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:08.427812 26512 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:08.427959 26512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:08.428122 26512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.8
I20240502 08:17:08.437494 26512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.207.8
--webserver_port=0
--tserver_master_addrs=127.20.207.62:36471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.8
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:08.439049 26512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:08.440830 26512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:08.453325 26520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:08.453828 26518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:08.454272 26517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:08.455328 26512 server_base.cc:1034] running on GCE node
I20240502 08:17:08.455958 26512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:08.456925 26512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:08.458091 26512 hybrid_clock.cc:648] HybridClock initialized: now 1714637828458033 us; error 46 us; skew 500 ppm
I20240502 08:17:08.458420 26512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:08.460703 26512 webserver.cc:460] Webserver started at http://127.20.207.8:43219/ using document root <none> and password file <none>
I20240502 08:17:08.461445 26512 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:08.461565 26512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:08.461880 26512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:08.464603 26512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "29f6d0e5c41046939872e76653b2be82"
format_stamp: "Formatted at 2024-05-02 08:17:08 on dist-test-slave-169q"
I20240502 08:17:08.465400 26512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "29f6d0e5c41046939872e76653b2be82"
format_stamp: "Formatted at 2024-05-02 08:17:08 on dist-test-slave-169q"
I20240502 08:17:08.470516 26512 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 08:17:08.473863 26526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:08.475175 26512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:17:08.475361 26512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "29f6d0e5c41046939872e76653b2be82"
format_stamp: "Formatted at 2024-05-02 08:17:08 on dist-test-slave-169q"
I20240502 08:17:08.475551 26512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:08.502686 26512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:08.503563 26512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:08.503861 26512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:08.504818 26512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:08.506311 26512 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:08.506413 26512 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:08.506541 26512 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:08.506613 26512 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:08.538309 26512 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.8:33143
I20240502 08:17:08.538367 26638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.8:33143 every 8 connection(s)
I20240502 08:17:08.539758 26512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 08:17:08.549775 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 26512
I20240502 08:17:08.550036 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 08:17:08.553373 26639 heartbeater.cc:347] Connected to a master server at 127.20.207.62:36471
I20240502 08:17:08.553792 26639 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:08.554903 26639 heartbeater.cc:511] Master 127.20.207.62:36471 requested a full tablet report, sending...
I20240502 08:17:08.558470 23546 ts_manager.cc:194] Registered new tserver with Master: 29f6d0e5c41046939872e76653b2be82 (127.20.207.8:33143)
I20240502 08:17:08.559711 23546 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.8:32893
I20240502 08:17:09.017848 23675 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:09.028340 24901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:09.037825 26574 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:09.047003 25293 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:09.095198 25032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:09.100790 25162 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:09.113188 25425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:09.125432 23808 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:09.289426 26574 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:09.305613 24901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:09.308897 23675 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:09.321429 25032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:09.322026 25293 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:09.342619 25162 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:09.354458 25425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:09.363692 23808 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:09.439810 23828 consensus_queue.cc:237] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: true } }
I20240502 08:17:09.441793 26731 raft_consensus.cc:2932] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29f6d0e5c41046939872e76653b2be82 (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: true } } }
I20240502 08:17:09.447214 23548 catalog_manager.cc:5484] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: config changed from index -1 to 4, NON_VOTER 29f6d0e5c41046939872e76653b2be82 (127.20.207.8) added. New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:09.453354 23760 consensus_peers.cc:485] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 -> Peer 29f6d0e5c41046939872e76653b2be82 (127.20.207.8:33143): Couldn't send request to peer 29f6d0e5c41046939872e76653b2be82. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac96d624768a4d26bc2cbc43d120d3d1. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:09.459971 23696 consensus_queue.cc:237] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: true } }
I20240502 08:17:09.461742 26744 raft_consensus.cc:2932] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29f6d0e5c41046939872e76653b2be82 (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: true } } }
I20240502 08:17:09.467244 23544 catalog_manager.cc:5484] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: config changed from index -1 to 4, NON_VOTER 29f6d0e5c41046939872e76653b2be82 (127.20.207.8) added. New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:09.470129 23629 consensus_peers.cc:485] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 -> Peer 29f6d0e5c41046939872e76653b2be82 (127.20.207.8:33143): Couldn't send request to peer 29f6d0e5c41046939872e76653b2be82. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b15e8c74b3b448538e315598c323601a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:09.562299 26639 heartbeater.cc:503] Master 127.20.207.62:36471 was elected leader, sending a full tablet report...
I20240502 08:17:10.005190 26778 ts_tablet_manager.cc:902] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82: Initiating tablet copy from peer 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519)
I20240502 08:17:10.006834 26778 tablet_copy_client.cc:308] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:41519
I20240502 08:17:10.012217 23716 tablet_copy_service.cc:140] P 3de510fd6a8a47a6acc24acba19b2364: Received BeginTabletCopySession request for tablet b15e8c74b3b448538e315598c323601a from peer 29f6d0e5c41046939872e76653b2be82 ({username='slave'} at 127.20.207.8:58173)
I20240502 08:17:10.012353 23716 tablet_copy_service.cc:161] P 3de510fd6a8a47a6acc24acba19b2364: Beginning new tablet copy session on tablet b15e8c74b3b448538e315598c323601a from peer 29f6d0e5c41046939872e76653b2be82 at {username='slave'} at 127.20.207.8:58173: session id = 29f6d0e5c41046939872e76653b2be82-b15e8c74b3b448538e315598c323601a
I20240502 08:17:10.014277 23716 tablet_copy_source_session.cc:205] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:10.018452 26778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15e8c74b3b448538e315598c323601a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:10.018841 26786 ts_tablet_manager.cc:902] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82: Initiating tablet copy from peer dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263)
I20240502 08:17:10.020603 26786 tablet_copy_client.cc:308] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.2:35263
I20240502 08:17:10.026743 23849 tablet_copy_service.cc:140] P dd7173a12d4a43b7bc728c0d99db5384: Received BeginTabletCopySession request for tablet ac96d624768a4d26bc2cbc43d120d3d1 from peer 29f6d0e5c41046939872e76653b2be82 ({username='slave'} at 127.20.207.8:34037)
I20240502 08:17:10.026908 23849 tablet_copy_service.cc:161] P dd7173a12d4a43b7bc728c0d99db5384: Beginning new tablet copy session on tablet ac96d624768a4d26bc2cbc43d120d3d1 from peer 29f6d0e5c41046939872e76653b2be82 at {username='slave'} at 127.20.207.8:34037: session id = 29f6d0e5c41046939872e76653b2be82-ac96d624768a4d26bc2cbc43d120d3d1
I20240502 08:17:10.029047 23849 tablet_copy_source_session.cc:205] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:10.029903 26778 tablet_copy_client.cc:783] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:10.030218 26778 tablet_copy_client.cc:647] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:10.030385 26786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac96d624768a4d26bc2cbc43d120d3d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:10.032621 26778 tablet_copy_client.cc:523] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:10.035465 26786 tablet_copy_client.cc:783] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:10.035691 26786 tablet_copy_client.cc:647] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:10.035766 26778 tablet_bootstrap.cc:492] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82: Bootstrap starting.
I20240502 08:17:10.037434 26786 tablet_copy_client.cc:523] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:10.039791 26786 tablet_bootstrap.cc:492] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82: Bootstrap starting.
I20240502 08:17:10.045447 26786 log.cc:824] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:10.075642 26786 tablet_bootstrap.cc:492] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:10.076153 26786 tablet_bootstrap.cc:492] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82: Bootstrap complete.
I20240502 08:17:10.076179 26778 tablet_bootstrap.cc:492] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:10.076535 26778 tablet_bootstrap.cc:492] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82: Bootstrap complete.
I20240502 08:17:10.076633 26786 ts_tablet_manager.cc:1367] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82: Time spent bootstrapping tablet: real 0.037s user 0.028s sys 0.006s
I20240502 08:17:10.076788 26778 ts_tablet_manager.cc:1367] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82: Time spent bootstrapping tablet: real 0.041s user 0.030s sys 0.009s
I20240502 08:17:10.081449 26786 raft_consensus.cc:348] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: true } }
I20240502 08:17:10.081442 26778 raft_consensus.cc:348] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: true } }
I20240502 08:17:10.081856 26786 raft_consensus.cc:729] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29f6d0e5c41046939872e76653b2be82, State: Initialized, Role: LEARNER
I20240502 08:17:10.081861 26778 raft_consensus.cc:729] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29f6d0e5c41046939872e76653b2be82, State: Initialized, Role: LEARNER
I20240502 08:17:10.082340 26778 consensus_queue.cc:260] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: true } }
I20240502 08:17:10.082343 26786 consensus_queue.cc:260] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: true } }
I20240502 08:17:10.084026 26786 ts_tablet_manager.cc:1398] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240502 08:17:10.084048 26778 ts_tablet_manager.cc:1398] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82: Time spent starting tablet: real 0.007s user 0.005s sys 0.004s
I20240502 08:17:10.085024 23716 tablet_copy_service.cc:342] P 3de510fd6a8a47a6acc24acba19b2364: Request end of tablet copy session 29f6d0e5c41046939872e76653b2be82-b15e8c74b3b448538e315598c323601a received from {username='slave'} at 127.20.207.8:58173
I20240502 08:17:10.085027 23849 tablet_copy_service.cc:342] P dd7173a12d4a43b7bc728c0d99db5384: Request end of tablet copy session 29f6d0e5c41046939872e76653b2be82-ac96d624768a4d26bc2cbc43d120d3d1 received from {username='slave'} at 127.20.207.8:34037
I20240502 08:17:10.085143 23716 tablet_copy_service.cc:433] P 3de510fd6a8a47a6acc24acba19b2364: ending tablet copy session 29f6d0e5c41046939872e76653b2be82-b15e8c74b3b448538e315598c323601a on tablet b15e8c74b3b448538e315598c323601a with peer 29f6d0e5c41046939872e76653b2be82
I20240502 08:17:10.085150 23849 tablet_copy_service.cc:433] P dd7173a12d4a43b7bc728c0d99db5384: ending tablet copy session 29f6d0e5c41046939872e76653b2be82-ac96d624768a4d26bc2cbc43d120d3d1 on tablet ac96d624768a4d26bc2cbc43d120d3d1 with peer 29f6d0e5c41046939872e76653b2be82
I20240502 08:17:10.469408 26594 raft_consensus.cc:1203] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:10.598650 26594 raft_consensus.cc:1203] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:10.874994 26816 raft_consensus.cc:1050] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364: attempting to promote NON_VOTER 29f6d0e5c41046939872e76653b2be82 to VOTER
I20240502 08:17:10.875620 26816 consensus_queue.cc:237] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } }
I20240502 08:17:10.877421 26594 raft_consensus.cc:1261] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 [term 1 LEARNER]: Refusing update from remote peer 3de510fd6a8a47a6acc24acba19b2364: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:10.878021 26816 consensus_queue.cc:1035] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:10.881314 26815 raft_consensus.cc:2932] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29f6d0e5c41046939872e76653b2be82 (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } } }
I20240502 08:17:10.881978 26594 raft_consensus.cc:2932] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29f6d0e5c41046939872e76653b2be82 (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } } }
I20240502 08:17:10.885596 23548 catalog_manager.cc:5484] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 reported cstate change: config changed from index 4 to 5, 29f6d0e5c41046939872e76653b2be82 (127.20.207.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3de510fd6a8a47a6acc24acba19b2364" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:11.011350 23696 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b15e8c74b3b448538e315598c323601a"
dest_uuid: "3de510fd6a8a47a6acc24acba19b2364"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54590
I20240502 08:17:11.011551 23696 raft_consensus.cc:595] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:11.101258 26829 raft_consensus.cc:1050] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384: attempting to promote NON_VOTER 29f6d0e5c41046939872e76653b2be82 to VOTER
I20240502 08:17:11.101806 26829 consensus_queue.cc:237] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } }
I20240502 08:17:11.103564 26594 raft_consensus.cc:1261] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 [term 1 LEARNER]: Refusing update from remote peer dd7173a12d4a43b7bc728c0d99db5384: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:11.104050 26829 consensus_queue.cc:1035] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:11.106354 26862 raft_consensus.cc:2932] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29f6d0e5c41046939872e76653b2be82 (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } } }
I20240502 08:17:11.106942 26594 raft_consensus.cc:2932] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29f6d0e5c41046939872e76653b2be82 (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } } }
I20240502 08:17:11.111729 23544 catalog_manager.cc:5484] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 reported cstate change: config changed from index 4 to 5, 29f6d0e5c41046939872e76653b2be82 (127.20.207.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7173a12d4a43b7bc728c0d99db5384" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:11.241236 23696 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b15e8c74b3b448538e315598c323601a"
dest_uuid: "3de510fd6a8a47a6acc24acba19b2364"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54616
I20240502 08:17:11.241448 23696 raft_consensus.cc:595] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:11.258265 23828 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac96d624768a4d26bc2cbc43d120d3d1"
dest_uuid: "dd7173a12d4a43b7bc728c0d99db5384"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37860
I20240502 08:17:11.258462 23828 raft_consensus.cc:595] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:11.307022 26816 raft_consensus.cc:979] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364: : Instructing follower 29f6d0e5c41046939872e76653b2be82 to start an election
I20240502 08:17:11.307193 26816 raft_consensus.cc:1067] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Signalling peer 29f6d0e5c41046939872e76653b2be82 to start an election
I20240502 08:17:11.308015 26594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b15e8c74b3b448538e315598c323601a"
dest_uuid: "29f6d0e5c41046939872e76653b2be82"
from {username='slave'} at 127.20.207.1:54467
I20240502 08:17:11.308280 26594 raft_consensus.cc:482] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:11.308450 26594 raft_consensus.cc:3037] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:11.310626 26594 raft_consensus.cc:504] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } }
I20240502 08:17:11.311810 26594 leader_election.cc:288] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 [CANDIDATE]: Term 2 election: Requested vote from peers 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519)
I20240502 08:17:11.312253 23696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15e8c74b3b448538e315598c323601a" candidate_uuid: "29f6d0e5c41046939872e76653b2be82" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3de510fd6a8a47a6acc24acba19b2364"
I20240502 08:17:11.312435 23696 raft_consensus.cc:3032] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:11.312521 23696 raft_consensus.cc:729] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3de510fd6a8a47a6acc24acba19b2364, State: Running, Role: LEADER
I20240502 08:17:11.312791 23696 consensus_queue.cc:260] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } }
I20240502 08:17:11.313024 23696 raft_consensus.cc:3037] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:11.314952 23696 raft_consensus.cc:2454] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f6d0e5c41046939872e76653b2be82 in term 2.
I20240502 08:17:11.315475 26528 leader_election.cc:302] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 29f6d0e5c41046939872e76653b2be82, 3de510fd6a8a47a6acc24acba19b2364; no voters:
I20240502 08:17:11.315794 26794 raft_consensus.cc:2781] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:11.316172 26794 raft_consensus.cc:686] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 [term 2 LEADER]: Becoming Leader. State: Replica: 29f6d0e5c41046939872e76653b2be82, State: Running, Role: LEADER
I20240502 08:17:11.316561 26794 consensus_queue.cc:237] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } }
I20240502 08:17:11.319346 23548 catalog_manager.cc:5484] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 reported cstate change: term changed from 1 to 2, leader changed from 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1) to 29f6d0e5c41046939872e76653b2be82 (127.20.207.8). New cstate: current_term: 2 leader_uuid: "29f6d0e5c41046939872e76653b2be82" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 08:17:11.328191 23537 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet b15e8c74b3b448538e315598c323601a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 29f6d0e5c41046939872e76653b2be82 (127.20.207.8:33143) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 08:17:11.482944 23828 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac96d624768a4d26bc2cbc43d120d3d1"
dest_uuid: "dd7173a12d4a43b7bc728c0d99db5384"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37882
I20240502 08:17:11.483114 23828 raft_consensus.cc:595] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:11.633496 26905 raft_consensus.cc:979] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384: : Instructing follower 29f6d0e5c41046939872e76653b2be82 to start an election
I20240502 08:17:11.633652 26905 raft_consensus.cc:1067] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Signalling peer 29f6d0e5c41046939872e76653b2be82 to start an election
I20240502 08:17:11.634335 26594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ac96d624768a4d26bc2cbc43d120d3d1"
dest_uuid: "29f6d0e5c41046939872e76653b2be82"
from {username='slave'} at 127.20.207.2:37877
I20240502 08:17:11.634528 26594 raft_consensus.cc:482] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:11.634639 26594 raft_consensus.cc:3037] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:11.636500 26594 raft_consensus.cc:504] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } }
I20240502 08:17:11.637045 26594 leader_election.cc:288] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 [CANDIDATE]: Term 2 election: Requested vote from peers dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263)
I20240502 08:17:11.637552 23828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac96d624768a4d26bc2cbc43d120d3d1" candidate_uuid: "29f6d0e5c41046939872e76653b2be82" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "dd7173a12d4a43b7bc728c0d99db5384"
I20240502 08:17:11.637744 23828 raft_consensus.cc:3032] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:11.637841 23828 raft_consensus.cc:729] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7173a12d4a43b7bc728c0d99db5384, State: Running, Role: LEADER
I20240502 08:17:11.638140 23828 consensus_queue.cc:260] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } }
I20240502 08:17:11.638482 23828 raft_consensus.cc:3037] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:11.640620 23828 raft_consensus.cc:2454] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f6d0e5c41046939872e76653b2be82 in term 2.
I20240502 08:17:11.641005 26530 leader_election.cc:302] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 29f6d0e5c41046939872e76653b2be82, dd7173a12d4a43b7bc728c0d99db5384; no voters:
I20240502 08:17:11.641261 26794 raft_consensus.cc:2781] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:11.641404 26794 raft_consensus.cc:686] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 [term 2 LEADER]: Becoming Leader. State: Replica: 29f6d0e5c41046939872e76653b2be82, State: Running, Role: LEADER
I20240502 08:17:11.641630 26794 consensus_queue.cc:237] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } }
I20240502 08:17:11.643579 23548 catalog_manager.cc:5484] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 reported cstate change: term changed from 1 to 2, leader changed from dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2) to 29f6d0e5c41046939872e76653b2be82 (127.20.207.8). New cstate: current_term: 2 leader_uuid: "29f6d0e5c41046939872e76653b2be82" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:11.736850 23696 raft_consensus.cc:1261] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Refusing update from remote peer 29f6d0e5c41046939872e76653b2be82: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:17:11.737569 26794 consensus_queue.cc:1035] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3de510fd6a8a47a6acc24acba19b2364" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 41519 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:11.953043 26594 consensus_queue.cc:237] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } }
I20240502 08:17:11.953984 26794 raft_consensus.cc:2932] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } } }
I20240502 08:17:11.957017 23537 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b15e8c74b3b448538e315598c323601a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 08:17:11.958371 23544 catalog_manager.cc:5484] T b15e8c74b3b448538e315598c323601a P 29f6d0e5c41046939872e76653b2be82 reported cstate change: config changed from index 5 to 7, VOTER 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1) evicted. New cstate: current_term: 2 leader_uuid: "29f6d0e5c41046939872e76653b2be82" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:11.961457 23675 tablet_service.cc:1514] Processing DeleteTablet for tablet b15e8c74b3b448538e315598c323601a with delete_type TABLET_DATA_TOMBSTONED (TS 3de510fd6a8a47a6acc24acba19b2364 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
I20240502 08:17:11.962076 26928 tablet_replica.cc:330] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364: stopping tablet replica
I20240502 08:17:11.962227 26928 raft_consensus.cc:2229] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:11.962380 26928 raft_consensus.cc:2258] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:11.963747 26928 ts_tablet_manager.cc:1875] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:11.968034 26928 ts_tablet_manager.cc:1888] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:17:11.968137 26928 log.cc:1196] T b15e8c74b3b448538e315598c323601a P 3de510fd6a8a47a6acc24acba19b2364: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/b15e8c74b3b448538e315598c323601a
I20240502 08:17:11.968869 23538 catalog_manager.cc:4830] TS 3de510fd6a8a47a6acc24acba19b2364 (127.20.207.1:41519): tablet b15e8c74b3b448538e315598c323601a (table rebalance_test_table_4 [id=ca0b1080aa984ac38b4aa6099730ddce]) successfully deleted
I20240502 08:17:12.259929 23828 raft_consensus.cc:1261] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Refusing update from remote peer 29f6d0e5c41046939872e76653b2be82: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:17:12.260538 26794 consensus_queue.cc:1035] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7173a12d4a43b7bc728c0d99db5384" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 35263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:12.269946 26594 consensus_queue.cc:237] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } }
I20240502 08:17:12.270731 26886 raft_consensus.cc:2932] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } } }
I20240502 08:17:12.273221 23537 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac96d624768a4d26bc2cbc43d120d3d1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 08:17:12.274354 23548 catalog_manager.cc:5484] T ac96d624768a4d26bc2cbc43d120d3d1 P 29f6d0e5c41046939872e76653b2be82 reported cstate change: config changed from index 5 to 7, VOTER dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2) evicted. New cstate: current_term: 2 leader_uuid: "29f6d0e5c41046939872e76653b2be82" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "29f6d0e5c41046939872e76653b2be82" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 33143 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:12.277300 23808 tablet_service.cc:1514] Processing DeleteTablet for tablet ac96d624768a4d26bc2cbc43d120d3d1 with delete_type TABLET_DATA_TOMBSTONED (TS dd7173a12d4a43b7bc728c0d99db5384 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49554
I20240502 08:17:12.277868 26948 tablet_replica.cc:330] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384: stopping tablet replica
I20240502 08:17:12.278079 26948 raft_consensus.cc:2229] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:12.278290 26948 raft_consensus.cc:2258] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:12.279810 26948 ts_tablet_manager.cc:1875] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:12.283972 26948 ts_tablet_manager.cc:1888] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:17:12.284075 26948 log.cc:1196] T ac96d624768a4d26bc2cbc43d120d3d1 P dd7173a12d4a43b7bc728c0d99db5384: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/ac96d624768a4d26bc2cbc43d120d3d1
I20240502 08:17:12.284837 23540 catalog_manager.cc:4830] TS dd7173a12d4a43b7bc728c0d99db5384 (127.20.207.2:35263): tablet ac96d624768a4d26bc2cbc43d120d3d1 (table rebalance_test_table_3 [id=0a34792f1dc944fea6b6ce2a62a12693]) successfully deleted
I20240502 08:17:12.464380 26574 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.481128 24901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.487257 25293 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.492900 23675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.515889 25032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.530661 25162 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.557862 25425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.591742 23808 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.721242 26574 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.726759 25293 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.742972 25032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.756122 24901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.758432 23675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.771046 25162 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.773200 25425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.777603 23808 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.063930 26574 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.065893 24901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.083652 23675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.100730 25162 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.110590 25032 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.112280 25293 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.121907 25425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.144711 23808 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
7eb46d820d5744549fe812ddd6fca0b0 | 127.20.207.62:36471 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.20.207.20:37025 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
29f6d0e5c41046939872e76653b2be82 | 127.20.207.8:33143 | HEALTHY | <none> | 2 | 0
39c4a2d0ccc94109b6bd2cac1792003f | 127.20.207.3:45649 | HEALTHY | <none> | 2 | 0
3de510fd6a8a47a6acc24acba19b2364 | 127.20.207.1:41519 | HEALTHY | <none> | 2 | 0
4b439f9c3d53400e8e2dc7b5c0a724b4 | 127.20.207.6:42387 | HEALTHY | <none> | 2 | 0
67e133ffa16d4a7a9ce63f934ab783b0 | 127.20.207.4:43335 | HEALTHY | <none> | 2 | 0
962f20ef8dea4cac91c4bf7603a655c0 | 127.20.207.5:44257 | HEALTHY | <none> | 2 | 0
cfb4a4db281241f5940203fe907dea50 | 127.20.207.7:39999 | HEALTHY | <none> | 2 | 0
dd7173a12d4a43b7bc728c0d99db5384 | 127.20.207.2:35263 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.20.207.1 | experimental | 127.20.207.1:41519
local_ip_for_outbound_sockets | 127.20.207.2 | experimental | 127.20.207.2:35263
local_ip_for_outbound_sockets | 127.20.207.3 | experimental | 127.20.207.3:45649
local_ip_for_outbound_sockets | 127.20.207.4 | experimental | 127.20.207.4:43335
local_ip_for_outbound_sockets | 127.20.207.5 | experimental | 127.20.207.5:44257
local_ip_for_outbound_sockets | 127.20.207.6 | experimental | 127.20.207.6:42387
local_ip_for_outbound_sockets | 127.20.207.7 | experimental | 127.20.207.7:39999
local_ip_for_outbound_sockets | 127.20.207.8 | experimental | 127.20.207.8:33143
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.20.207.1:41519
server_dump_info_path | /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.20.207.2:35263
server_dump_info_path | /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.20.207.3:45649
server_dump_info_path | /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.20.207.4:43335
server_dump_info_path | /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.20.207.5:44257
server_dump_info_path | /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.20.207.6:42387
server_dump_info_path | /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.20.207.7:39999
server_dump_info_path | /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.20.207.8:33143
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.20.207.20:37025 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 08:17:13.200148 21308 log_verifier.cc:126] Checking tablet 1c26c8082287402daf70ee9be5a4fb7f
I20240502 08:17:13.210748 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 1c26c8082287402daf70ee9be5a4fb7f
I20240502 08:17:13.210865 21308 log_verifier.cc:126] Checking tablet 2c0157c3916c4012a6b8c4d38df7a342
I20240502 08:17:13.221694 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 2c0157c3916c4012a6b8c4d38df7a342
I20240502 08:17:13.221802 21308 log_verifier.cc:126] Checking tablet 7df5ab24ba3c42769dd4640220641996
I20240502 08:17:13.232229 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 7df5ab24ba3c42769dd4640220641996
I20240502 08:17:13.232333 21308 log_verifier.cc:126] Checking tablet 8096d61335f8493594e55720d47489bd
I20240502 08:17:13.241986 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 8096d61335f8493594e55720d47489bd
I20240502 08:17:13.242130 21308 log_verifier.cc:126] Checking tablet 9506dfea92e3488fbe504216a75c4798
I20240502 08:17:13.251281 21308 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9506dfea92e3488fbe504216a75c4798
I20240502 08:17:13.251375 21308 log_verifier.cc:126] Checking tablet 9a4dfe88384b446fa08524fa91e557b7
I20240502 08:17:13.260906 21308 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9a4dfe88384b446fa08524fa91e557b7
I20240502 08:17:13.261016 21308 log_verifier.cc:126] Checking tablet ac96d624768a4d26bc2cbc43d120d3d1
I20240502 08:17:13.276947 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet ac96d624768a4d26bc2cbc43d120d3d1
I20240502 08:17:13.277069 21308 log_verifier.cc:126] Checking tablet b15e8c74b3b448538e315598c323601a
I20240502 08:17:13.293593 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet b15e8c74b3b448538e315598c323601a
I20240502 08:17:13.293694 21308 log_verifier.cc:126] Checking tablet bf84e851bd9e41acb2fc657fb516cb6e
I20240502 08:17:13.302958 21308 log_verifier.cc:177] Verified matching terms for 3 ops in tablet bf84e851bd9e41acb2fc657fb516cb6e
I20240502 08:17:13.303045 21308 log_verifier.cc:126] Checking tablet c9d6f55ce8f547e395d686838dbb7bd2
I20240502 08:17:13.312618 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c9d6f55ce8f547e395d686838dbb7bd2
I20240502 08:17:13.312719 21308 log_verifier.cc:126] Checking tablet ce947f6507ea457da00437f718444e5b
I20240502 08:17:13.322610 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet ce947f6507ea457da00437f718444e5b
I20240502 08:17:13.322710 21308 log_verifier.cc:126] Checking tablet db6049ba7a5640d4ac4dae656b2f3667
I20240502 08:17:13.332965 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet db6049ba7a5640d4ac4dae656b2f3667
I20240502 08:17:13.333073 21308 log_verifier.cc:126] Checking tablet eff43ec5b0ea4198ae23d57f2c4cb2c9
I20240502 08:17:13.338692 21308 log_verifier.cc:177] Verified matching terms for 5 ops in tablet eff43ec5b0ea4198ae23d57f2c4cb2c9
I20240502 08:17:13.338796 21308 log_verifier.cc:126] Checking tablet f214664b085a4ac0a10484acbd525142
I20240502 08:17:13.348495 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet f214664b085a4ac0a10484acbd525142
I20240502 08:17:13.348605 21308 log_verifier.cc:126] Checking tablet f6dd3894b6aa489f92fcb1aa383b2a58
I20240502 08:17:13.357932 21308 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f6dd3894b6aa489f92fcb1aa383b2a58
I20240502 08:17:13.358047 21308 log_verifier.cc:126] Checking tablet fb067e2ff17740958059cfef1e0d0f9d
I20240502 08:17:13.367893 21308 log_verifier.cc:177] Verified matching terms for 7 ops in tablet fb067e2ff17740958059cfef1e0d0f9d
I20240502 08:17:13.368786 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 23614
I20240502 08:17:13.370204 23736 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:13.506067 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 23614
I20240502 08:17:13.531329 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 23745
I20240502 08:17:13.532298 23869 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:13.669767 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 23745
I20240502 08:17:13.694656 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 24838
I20240502 08:17:13.700618 24961 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:13.813261 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 24838
I20240502 08:17:13.833797 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 24970
I20240502 08:17:13.839466 25092 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:13.960094 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 24970
I20240502 08:17:13.980480 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 25100
I20240502 08:17:13.986615 25222 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:14.100894 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 25100
I20240502 08:17:14.121513 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 25230
I20240502 08:17:14.127723 25353 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:14.240743 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 25230
I20240502 08:17:14.261147 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 25362
I20240502 08:17:14.266750 25484 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:14.379046 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 25362
W20240502 08:17:14.399605 23540 connection.cc:535] server connection from 127.20.207.7:40599 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:14.399832 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 26512
I20240502 08:17:14.405467 26634 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:14.505030 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 26512
I20240502 08:17:14.523537 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 23522
I20240502 08:17:14.524588 23583 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:14.671094 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 23522
2024-05-02T08:17:14Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/1 (26825 ms)
[----------] 1 test from RebalanceParamTest (26825 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1
I20240502 08:17:14.740365 21308 test_util.cc:268] Using random seed: 990831219
I20240502 08:17:14.742420 21308 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:17:14.742486 21308 ts_itest-base.cc:113] --------------
I20240502 08:17:14.742535 21308 ts_itest-base.cc:114] 8 tablet servers
I20240502 08:17:14.742583 21308 ts_itest-base.cc:115] 3 replicas per TS
I20240502 08:17:14.742647 21308 ts_itest-base.cc:116] --------------
2024-05-02T08:17:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:17:14Z Disabled control of system clock
I20240502 08:17:14.770964 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.207.62:41499
--webserver_interface=127.20.207.62
--webserver_port=0
--builtin_ntp_servers=127.20.207.20:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.207.62:41499
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 08:17:15.089116 27085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:15.089421 27085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:15.089504 27085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:15.096572 27085 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:15.096688 27085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:17:15.096740 27085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:15.096794 27085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:17:15.096838 27085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:17:15.105496 27085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.207.62:41499
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.207.62:41499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.207.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:15.106774 27085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:15.108511 27085 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:15.118754 27090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:15.119464 27091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:15.120385 27093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:15.121095 27085 server_base.cc:1034] running on GCE node
I20240502 08:17:15.121814 27085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:15.123239 27085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:15.124447 27085 hybrid_clock.cc:648] HybridClock initialized: now 1714637835124402 us; error 34 us; skew 500 ppm
I20240502 08:17:15.124785 27085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:15.127019 27085 webserver.cc:460] Webserver started at http://127.20.207.62:34823/ using document root <none> and password file <none>
I20240502 08:17:15.127759 27085 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:15.127864 27085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:15.128171 27085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:15.130795 27085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "c086547273ea42f0a31343f9f60d13f0"
format_stamp: "Formatted at 2024-05-02 08:17:15 on dist-test-slave-169q"
I20240502 08:17:15.131587 27085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "c086547273ea42f0a31343f9f60d13f0"
format_stamp: "Formatted at 2024-05-02 08:17:15 on dist-test-slave-169q"
I20240502 08:17:15.136586 27085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:17:15.139863 27099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:15.141064 27085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:15.141222 27085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
uuid: "c086547273ea42f0a31343f9f60d13f0"
format_stamp: "Formatted at 2024-05-02 08:17:15 on dist-test-slave-169q"
I20240502 08:17:15.141412 27085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:15.168190 27085 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:15.169016 27085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:15.169288 27085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:15.186398 27085 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.62:41499
I20240502 08:17:15.186405 27151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.62:41499 every 8 connection(s)
I20240502 08:17:15.187729 27085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:17:15.189776 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 27085
I20240502 08:17:15.189975 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:17:15.192157 27152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:15.205802 27152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Bootstrap starting.
I20240502 08:17:15.208554 27152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:15.209509 27152 log.cc:824] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:15.212013 27152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: No bootstrap required, opened a new log
I20240502 08:17:15.218154 27152 raft_consensus.cc:348] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c086547273ea42f0a31343f9f60d13f0" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 41499 } }
I20240502 08:17:15.218485 27152 raft_consensus.cc:374] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:15.218561 27152 raft_consensus.cc:729] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c086547273ea42f0a31343f9f60d13f0, State: Initialized, Role: FOLLOWER
I20240502 08:17:15.219115 27152 consensus_queue.cc:260] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c086547273ea42f0a31343f9f60d13f0" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 41499 } }
I20240502 08:17:15.219288 27152 raft_consensus.cc:388] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:17:15.219377 27152 raft_consensus.cc:482] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:17:15.219496 27152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:15.221588 27152 raft_consensus.cc:504] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c086547273ea42f0a31343f9f60d13f0" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 41499 } }
I20240502 08:17:15.222198 27152 leader_election.cc:302] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c086547273ea42f0a31343f9f60d13f0; no voters:
I20240502 08:17:15.222929 27152 leader_election.cc:288] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:17:15.223181 27157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:15.224992 27157 raft_consensus.cc:686] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0 [term 1 LEADER]: Becoming Leader. State: Replica: c086547273ea42f0a31343f9f60d13f0, State: Running, Role: LEADER
I20240502 08:17:15.225457 27157 consensus_queue.cc:237] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c086547273ea42f0a31343f9f60d13f0" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 41499 } }
I20240502 08:17:15.225832 27152 sys_catalog.cc:564] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:17:15.228000 27158 sys_catalog.cc:455] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c086547273ea42f0a31343f9f60d13f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c086547273ea42f0a31343f9f60d13f0" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 41499 } } }
I20240502 08:17:15.228024 27159 sys_catalog.cc:455] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c086547273ea42f0a31343f9f60d13f0. Latest consensus state: current_term: 1 leader_uuid: "c086547273ea42f0a31343f9f60d13f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c086547273ea42f0a31343f9f60d13f0" member_type: VOTER last_known_addr { host: "127.20.207.62" port: 41499 } } }
I20240502 08:17:15.228389 27158 sys_catalog.cc:458] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0 [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:15.228530 27159 sys_catalog.cc:458] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0 [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:15.230415 27162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:17:15.235567 27162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:17:15.247426 27162 catalog_manager.cc:1347] Generated new cluster ID: 1da64d1bf7694c07a07db9b3555accf5
I20240502 08:17:15.247545 27162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:17:15.272009 27162 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:17:15.273204 27162 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:17:15.281579 27162 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Generated new TSK 0
I20240502 08:17:15.282438 27162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:17:15.291000 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.1:0
--local_ip_for_outbound_sockets=127.20.207.1
--webserver_interface=127.20.207.1
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41499
--builtin_ntp_servers=127.20.207.20:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:17:15.623960 27176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:15.624251 27176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:15.624346 27176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:15.632012 27176 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:15.632129 27176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:15.632318 27176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.1
I20240502 08:17:15.641985 27176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.207.1
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.1
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:15.643354 27176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:15.645071 27176 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:15.657037 27184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:15.657641 27182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:15.658285 27181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:15.659140 27176 server_base.cc:1034] running on GCE node
I20240502 08:17:15.659719 27176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:15.660669 27176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:15.661878 27176 hybrid_clock.cc:648] HybridClock initialized: now 1714637835661833 us; error 41 us; skew 500 ppm
I20240502 08:17:15.662230 27176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:15.664434 27176 webserver.cc:460] Webserver started at http://127.20.207.1:36381/ using document root <none> and password file <none>
I20240502 08:17:15.665151 27176 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:15.665272 27176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:15.665602 27176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:15.668138 27176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
format_stamp: "Formatted at 2024-05-02 08:17:15 on dist-test-slave-169q"
I20240502 08:17:15.668881 27176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
format_stamp: "Formatted at 2024-05-02 08:17:15 on dist-test-slave-169q"
I20240502 08:17:15.673691 27176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 08:17:15.676885 27190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:15.678115 27176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240502 08:17:15.678337 27176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
format_stamp: "Formatted at 2024-05-02 08:17:15 on dist-test-slave-169q"
I20240502 08:17:15.678553 27176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:15.696537 27176 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:15.697383 27176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:15.697654 27176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:15.698647 27176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:15.700079 27176 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:15.700184 27176 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:15.700299 27176 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:15.700373 27176 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:15.731067 27176 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.1:38505
I20240502 08:17:15.731115 27302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.1:38505 every 8 connection(s)
I20240502 08:17:15.732338 27176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:17:15.734385 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 27176
I20240502 08:17:15.734630 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:17:15.738168 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.2:0
--local_ip_for_outbound_sockets=127.20.207.2
--webserver_interface=127.20.207.2
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41499
--builtin_ntp_servers=127.20.207.20:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:15.746075 27303 heartbeater.cc:347] Connected to a master server at 127.20.207.62:41499
I20240502 08:17:15.746390 27303 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:15.747181 27303 heartbeater.cc:511] Master 127.20.207.62:41499 requested a full tablet report, sending...
I20240502 08:17:15.749343 27116 ts_manager.cc:194] Registered new tserver with Master: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:15.751531 27116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.1:55803
W20240502 08:17:16.065788 27307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:16.066109 27307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:16.066215 27307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:16.073407 27307 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:16.073551 27307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:16.073674 27307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.2
I20240502 08:17:16.082408 27307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.207.2
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.2
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:16.083734 27307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:16.085460 27307 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:16.097404 27313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:16.097841 27312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:16.098591 27315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:16.099515 27307 server_base.cc:1034] running on GCE node
I20240502 08:17:16.100097 27307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:16.100955 27307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:16.102144 27307 hybrid_clock.cc:648] HybridClock initialized: now 1714637836102097 us; error 42 us; skew 500 ppm
I20240502 08:17:16.102442 27307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:16.110605 27307 webserver.cc:460] Webserver started at http://127.20.207.2:33579/ using document root <none> and password file <none>
I20240502 08:17:16.111356 27307 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:16.111469 27307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:16.111766 27307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:16.114364 27307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
format_stamp: "Formatted at 2024-05-02 08:17:16 on dist-test-slave-169q"
I20240502 08:17:16.115113 27307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
format_stamp: "Formatted at 2024-05-02 08:17:16 on dist-test-slave-169q"
I20240502 08:17:16.120209 27307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:17:16.123476 27322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:16.124730 27307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:16.124879 27307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
format_stamp: "Formatted at 2024-05-02 08:17:16 on dist-test-slave-169q"
I20240502 08:17:16.125072 27307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:16.158571 27307 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:16.159507 27307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:16.159770 27307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:16.160804 27307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:16.162216 27307 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:16.162307 27307 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:16.162386 27307 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:16.162437 27307 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:16.211726 27307 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.2:46291
I20240502 08:17:16.211791 27434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.2:46291 every 8 connection(s)
I20240502 08:17:16.213438 27307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:17:16.221354 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 27307
I20240502 08:17:16.221563 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:17:16.225872 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.3:0
--local_ip_for_outbound_sockets=127.20.207.3
--webserver_interface=127.20.207.3
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41499
--builtin_ntp_servers=127.20.207.20:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:16.227105 27435 heartbeater.cc:347] Connected to a master server at 127.20.207.62:41499
I20240502 08:17:16.227475 27435 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:16.228499 27435 heartbeater.cc:511] Master 127.20.207.62:41499 requested a full tablet report, sending...
I20240502 08:17:16.230132 27116 ts_manager.cc:194] Registered new tserver with Master: 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:16.231204 27116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.2:33235
W20240502 08:17:16.545857 27438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:16.546198 27438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:16.546321 27438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:16.553953 27438 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:16.554113 27438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:16.554252 27438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.3
I20240502 08:17:16.563458 27438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.207.3
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.3
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:16.564767 27438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:16.566581 27438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:16.578420 27446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:16.578907 27443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:16.579767 27444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:16.580652 27438 server_base.cc:1034] running on GCE node
I20240502 08:17:16.581178 27438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:16.582116 27438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:16.583302 27438 hybrid_clock.cc:648] HybridClock initialized: now 1714637836583246 us; error 56 us; skew 500 ppm
I20240502 08:17:16.583582 27438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:16.590943 27438 webserver.cc:460] Webserver started at http://127.20.207.3:46133/ using document root <none> and password file <none>
I20240502 08:17:16.591651 27438 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:16.591761 27438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:16.592051 27438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:16.594595 27438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b01e7c8a302d41078fa39cfef665ff59"
format_stamp: "Formatted at 2024-05-02 08:17:16 on dist-test-slave-169q"
I20240502 08:17:16.595351 27438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b01e7c8a302d41078fa39cfef665ff59"
format_stamp: "Formatted at 2024-05-02 08:17:16 on dist-test-slave-169q"
I20240502 08:17:16.600561 27438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:17:16.603783 27452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:16.605029 27438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:16.605190 27438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b01e7c8a302d41078fa39cfef665ff59"
format_stamp: "Formatted at 2024-05-02 08:17:16 on dist-test-slave-169q"
I20240502 08:17:16.605377 27438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:16.625515 27438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:16.626458 27438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:16.626720 27438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:16.627696 27438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:16.629132 27438 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:16.629231 27438 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:16.629343 27438 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:16.629413 27438 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:16.660763 27438 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.3:44469
I20240502 08:17:16.660802 27565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.3:44469 every 8 connection(s)
I20240502 08:17:16.662134 27438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:17:16.670066 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 27438
I20240502 08:17:16.670279 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:17:16.673553 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.4:0
--local_ip_for_outbound_sockets=127.20.207.4
--webserver_interface=127.20.207.4
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41499
--builtin_ntp_servers=127.20.207.20:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:16.674515 27566 heartbeater.cc:347] Connected to a master server at 127.20.207.62:41499
I20240502 08:17:16.674790 27566 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:16.675562 27566 heartbeater.cc:511] Master 127.20.207.62:41499 requested a full tablet report, sending...
I20240502 08:17:16.676986 27116 ts_manager.cc:194] Registered new tserver with Master: b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:16.678073 27116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.3:37071
I20240502 08:17:16.754427 27303 heartbeater.cc:503] Master 127.20.207.62:41499 was elected leader, sending a full tablet report...
W20240502 08:17:16.996836 27569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:16.997123 27569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:16.997288 27569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:17.004426 27569 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:17.004565 27569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:17.004731 27569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.4
I20240502 08:17:17.013700 27569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.207.4
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.4
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:17.015035 27569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:17.016722 27569 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:17.028327 27574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:17.029071 27577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:17.029827 27575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:17.030689 27569 server_base.cc:1034] running on GCE node
I20240502 08:17:17.031237 27569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:17.032079 27569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:17.033282 27569 hybrid_clock.cc:648] HybridClock initialized: now 1714637837033238 us; error 33 us; skew 500 ppm
I20240502 08:17:17.033565 27569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:17.035838 27569 webserver.cc:460] Webserver started at http://127.20.207.4:38005/ using document root <none> and password file <none>
I20240502 08:17:17.036588 27569 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:17.036692 27569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:17.036993 27569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:17.039502 27569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "bf8ac71b15f740a0a928911360773442"
format_stamp: "Formatted at 2024-05-02 08:17:17 on dist-test-slave-169q"
I20240502 08:17:17.040227 27569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "bf8ac71b15f740a0a928911360773442"
format_stamp: "Formatted at 2024-05-02 08:17:17 on dist-test-slave-169q"
I20240502 08:17:17.045017 27569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240502 08:17:17.048250 27583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:17.049428 27569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:17.049561 27569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "bf8ac71b15f740a0a928911360773442"
format_stamp: "Formatted at 2024-05-02 08:17:17 on dist-test-slave-169q"
I20240502 08:17:17.049736 27569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:17.075857 27569 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:17.076673 27569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:17.076922 27569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:17.077836 27569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:17.079355 27569 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:17.079442 27569 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:17.079543 27569 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:17.079612 27569 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:17.111029 27569 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.4:33309
I20240502 08:17:17.111095 27696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.4:33309 every 8 connection(s)
I20240502 08:17:17.112296 27569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:17:17.114392 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 27569
I20240502 08:17:17.114554 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:17:17.117935 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.5:0
--local_ip_for_outbound_sockets=127.20.207.5
--webserver_interface=127.20.207.5
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41499
--builtin_ntp_servers=127.20.207.20:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:17.125955 27697 heartbeater.cc:347] Connected to a master server at 127.20.207.62:41499
I20240502 08:17:17.126258 27697 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:17.127033 27697 heartbeater.cc:511] Master 127.20.207.62:41499 requested a full tablet report, sending...
I20240502 08:17:17.128425 27116 ts_manager.cc:194] Registered new tserver with Master: bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:17.129550 27116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.4:55659
I20240502 08:17:17.233667 27435 heartbeater.cc:503] Master 127.20.207.62:41499 was elected leader, sending a full tablet report...
W20240502 08:17:17.443940 27700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:17.444214 27700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:17.444350 27700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:17.451437 27700 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:17.451553 27700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:17.451705 27700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.5
I20240502 08:17:17.460472 27700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.207.5
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.5
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:17.461714 27700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:17.463491 27700 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:17.474978 27706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:17.475659 27705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:17.476369 27708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:17.477102 27700 server_base.cc:1034] running on GCE node
I20240502 08:17:17.477695 27700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:17.478700 27700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:17.479908 27700 hybrid_clock.cc:648] HybridClock initialized: now 1714637837479863 us; error 43 us; skew 500 ppm
I20240502 08:17:17.480227 27700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:17.482432 27700 webserver.cc:460] Webserver started at http://127.20.207.5:35485/ using document root <none> and password file <none>
I20240502 08:17:17.483130 27700 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:17.483281 27700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:17.483608 27700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:17.486109 27700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "1f0bf4bc3c5146539146b825b048bff3"
format_stamp: "Formatted at 2024-05-02 08:17:17 on dist-test-slave-169q"
I20240502 08:17:17.486898 27700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "1f0bf4bc3c5146539146b825b048bff3"
format_stamp: "Formatted at 2024-05-02 08:17:17 on dist-test-slave-169q"
I20240502 08:17:17.491645 27700 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:17:17.494933 27714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:17.496187 27700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 08:17:17.496351 27700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1f0bf4bc3c5146539146b825b048bff3"
format_stamp: "Formatted at 2024-05-02 08:17:17 on dist-test-slave-169q"
I20240502 08:17:17.496531 27700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:17.516631 27700 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:17.517467 27700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:17.517730 27700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:17.518673 27700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:17.520221 27700 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:17.520315 27700 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:17.520423 27700 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:17.520493 27700 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:17.551606 27700 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.5:37815
I20240502 08:17:17.551649 27828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.5:37815 every 8 connection(s)
I20240502 08:17:17.552879 27700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:17:17.559784 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 27700
I20240502 08:17:17.559990 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:17:17.563205 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.6:0
--local_ip_for_outbound_sockets=127.20.207.6
--webserver_interface=127.20.207.6
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41499
--builtin_ntp_servers=127.20.207.20:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:17.566076 27829 heartbeater.cc:347] Connected to a master server at 127.20.207.62:41499
I20240502 08:17:17.566365 27829 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:17.567281 27829 heartbeater.cc:511] Master 127.20.207.62:41499 requested a full tablet report, sending...
I20240502 08:17:17.568642 27116 ts_manager.cc:194] Registered new tserver with Master: 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:17:17.569393 27116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.5:34261
I20240502 08:17:17.680456 27566 heartbeater.cc:503] Master 127.20.207.62:41499 was elected leader, sending a full tablet report...
W20240502 08:17:17.883322 27832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:17.883605 27832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:17.883729 27832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:17.891021 27832 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:17.891137 27832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:17.891296 27832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.6
I20240502 08:17:17.900204 27832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.207.6
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.6
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:17.901459 27832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:17.903118 27832 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:17.916705 27837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:17.917405 27838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:17.918241 27840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:17.919250 27832 server_base.cc:1034] running on GCE node
I20240502 08:17:17.919943 27832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:17.920886 27832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:17.922073 27832 hybrid_clock.cc:648] HybridClock initialized: now 1714637837922023 us; error 50 us; skew 500 ppm
I20240502 08:17:17.922358 27832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:17.924494 27832 webserver.cc:460] Webserver started at http://127.20.207.6:37797/ using document root <none> and password file <none>
I20240502 08:17:17.925186 27832 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:17.925299 27832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:17.925597 27832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:17.928239 27832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "84cda20738d2452da166e31fc6da57a6"
format_stamp: "Formatted at 2024-05-02 08:17:17 on dist-test-slave-169q"
I20240502 08:17:17.929000 27832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "84cda20738d2452da166e31fc6da57a6"
format_stamp: "Formatted at 2024-05-02 08:17:17 on dist-test-slave-169q"
I20240502 08:17:17.933786 27832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20240502 08:17:17.937077 27846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:17.938374 27832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:17:17.938529 27832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "84cda20738d2452da166e31fc6da57a6"
format_stamp: "Formatted at 2024-05-02 08:17:17 on dist-test-slave-169q"
I20240502 08:17:17.938720 27832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:17.957099 27832 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:17.957899 27832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:17.958217 27832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:17.959153 27832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:17.960605 27832 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:17.960705 27832 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:17.960824 27832 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:17.960896 27832 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:17.992019 27832 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.6:38899
I20240502 08:17:17.992063 27958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.6:38899 every 8 connection(s)
I20240502 08:17:17.993343 27832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:17:17.994287 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 27832
I20240502 08:17:17.994453 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 08:17:17.997865 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.7:0
--local_ip_for_outbound_sockets=127.20.207.7
--webserver_interface=127.20.207.7
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41499
--builtin_ntp_servers=127.20.207.20:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:18.006734 27959 heartbeater.cc:347] Connected to a master server at 127.20.207.62:41499
I20240502 08:17:18.007020 27959 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:18.007804 27959 heartbeater.cc:511] Master 127.20.207.62:41499 requested a full tablet report, sending...
I20240502 08:17:18.009130 27116 ts_manager.cc:194] Registered new tserver with Master: 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:18.009938 27116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.6:35741
I20240502 08:17:18.132094 27697 heartbeater.cc:503] Master 127.20.207.62:41499 was elected leader, sending a full tablet report...
W20240502 08:17:18.322137 27962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:18.322417 27962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:18.322544 27962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:18.329747 27962 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:18.329870 27962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:18.330053 27962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.7
I20240502 08:17:18.338662 27962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.207.7
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.7
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:18.339946 27962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:18.341673 27962 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:18.353204 27968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:18.353881 27967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:18.354732 27970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:18.355715 27962 server_base.cc:1034] running on GCE node
I20240502 08:17:18.356292 27962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:18.357164 27962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:18.358376 27962 hybrid_clock.cc:648] HybridClock initialized: now 1714637838358340 us; error 31 us; skew 500 ppm
I20240502 08:17:18.358660 27962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:18.360777 27962 webserver.cc:460] Webserver started at http://127.20.207.7:44619/ using document root <none> and password file <none>
I20240502 08:17:18.361467 27962 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:18.361572 27962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:18.361860 27962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:18.364277 27962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
format_stamp: "Formatted at 2024-05-02 08:17:18 on dist-test-slave-169q"
I20240502 08:17:18.365061 27962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
format_stamp: "Formatted at 2024-05-02 08:17:18 on dist-test-slave-169q"
I20240502 08:17:18.369848 27962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240502 08:17:18.372895 27976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:18.374094 27962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240502 08:17:18.374236 27962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
format_stamp: "Formatted at 2024-05-02 08:17:18 on dist-test-slave-169q"
I20240502 08:17:18.374418 27962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:18.392053 27962 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:18.392901 27962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:18.393170 27962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:18.394109 27962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:18.395473 27962 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:18.395565 27962 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:18.395678 27962 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:18.395750 27962 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:18.426493 27962 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.7:40343
I20240502 08:17:18.426532 28088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.7:40343 every 8 connection(s)
I20240502 08:17:18.427806 27962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 08:17:18.428643 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 27962
I20240502 08:17:18.428838 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 08:17:18.432310 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.8:0
--local_ip_for_outbound_sockets=127.20.207.8
--webserver_interface=127.20.207.8
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41499
--builtin_ntp_servers=127.20.207.20:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:18.442103 28089 heartbeater.cc:347] Connected to a master server at 127.20.207.62:41499
I20240502 08:17:18.442406 28089 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:18.443181 28089 heartbeater.cc:511] Master 127.20.207.62:41499 requested a full tablet report, sending...
I20240502 08:17:18.444649 27116 ts_manager.cc:194] Registered new tserver with Master: 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:17:18.445763 27116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.7:40107
I20240502 08:17:18.572422 27829 heartbeater.cc:503] Master 127.20.207.62:41499 was elected leader, sending a full tablet report...
W20240502 08:17:18.754348 28092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:18.754623 28092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:18.754716 28092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:18.762251 28092 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:18.762367 28092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:18.762490 28092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.8
I20240502 08:17:18.771446 28092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.207.8
--webserver_port=0
--tserver_master_addrs=127.20.207.62:41499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.8
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:18.772753 28092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:18.774493 28092 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:18.786463 28097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:18.787209 28100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:18.787994 28098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:18.788707 28092 server_base.cc:1034] running on GCE node
I20240502 08:17:18.789265 28092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:18.790184 28092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:18.791381 28092 hybrid_clock.cc:648] HybridClock initialized: now 1714637838791330 us; error 41 us; skew 500 ppm
I20240502 08:17:18.791731 28092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:18.793869 28092 webserver.cc:460] Webserver started at http://127.20.207.8:45047/ using document root <none> and password file <none>
I20240502 08:17:18.794667 28092 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:18.794785 28092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:18.795074 28092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:18.797561 28092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
format_stamp: "Formatted at 2024-05-02 08:17:18 on dist-test-slave-169q"
I20240502 08:17:18.798344 28092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
format_stamp: "Formatted at 2024-05-02 08:17:18 on dist-test-slave-169q"
I20240502 08:17:18.803328 28092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 08:17:18.806625 28106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:18.807850 28092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:17:18.808002 28092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
format_stamp: "Formatted at 2024-05-02 08:17:18 on dist-test-slave-169q"
I20240502 08:17:18.808244 28092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:18.832504 28092 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:18.833338 28092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:18.833588 28092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:18.834614 28092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:18.836079 28092 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:18.836176 28092 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:18.836298 28092 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:18.836370 28092 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:18.867627 28092 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.8:41619
I20240502 08:17:18.867671 28219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.8:41619 every 8 connection(s)
I20240502 08:17:18.868908 28092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 08:17:18.874814 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 28092
I20240502 08:17:18.874994 21308 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 08:17:18.881516 28220 heartbeater.cc:347] Connected to a master server at 127.20.207.62:41499
I20240502 08:17:18.881788 28220 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:18.882735 28220 heartbeater.cc:511] Master 127.20.207.62:41499 requested a full tablet report, sending...
I20240502 08:17:18.884096 27116 ts_manager.cc:194] Registered new tserver with Master: 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:17:18.884832 27116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.8:39359
I20240502 08:17:18.890885 21308 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 08:17:18.907464 21308 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 08:17:18.907613 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 27700
I20240502 08:17:18.915851 27824 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:19.002946 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 27700
I20240502 08:17:19.012427 27959 heartbeater.cc:503] Master 127.20.207.62:41499 was elected leader, sending a full tablet report...
I20240502 08:17:19.019259 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 27832
I20240502 08:17:19.027266 27954 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:19.105213 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 27832
I20240502 08:17:19.121284 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 27962
I20240502 08:17:19.129104 28084 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:19.207434 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 27962
I20240502 08:17:19.223233 21308 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu pid 28092
I20240502 08:17:19.231565 28215 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:19.312130 21308 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu with pid 28092
I20240502 08:17:23.079653 27116 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:23.115243 27500 tablet_service.cc:1467] Processing CreateTablet for tablet 7f5a95b67d1c45578e323cc4ce8d4c14 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a7bef71aae7647a990f6008892e44102]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:23.116266 27499 tablet_service.cc:1467] Processing CreateTablet for tablet d465bff0ba8c42458e7308bb85adfc65 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a7bef71aae7647a990f6008892e44102]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:23.117029 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f5a95b67d1c45578e323cc4ce8d4c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:23.117560 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d465bff0ba8c42458e7308bb85adfc65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:23.123064 27630 tablet_service.cc:1467] Processing CreateTablet for tablet e15163867cbf4cc2ab2d6da3e430b261 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a7bef71aae7647a990f6008892e44102]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:23.124351 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e15163867cbf4cc2ab2d6da3e430b261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:23.126492 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 7f5a95b67d1c45578e323cc4ce8d4c14 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a7bef71aae7647a990f6008892e44102]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:23.126896 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f5a95b67d1c45578e323cc4ce8d4c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:23.126821 27370 tablet_service.cc:1467] Processing CreateTablet for tablet 7f5a95b67d1c45578e323cc4ce8d4c14 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a7bef71aae7647a990f6008892e44102]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:23.128072 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f5a95b67d1c45578e323cc4ce8d4c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:23.130339 27369 tablet_service.cc:1467] Processing CreateTablet for tablet d465bff0ba8c42458e7308bb85adfc65 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a7bef71aae7647a990f6008892e44102]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:23.130718 27369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d465bff0ba8c42458e7308bb85adfc65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:23.131350 27368 tablet_service.cc:1467] Processing CreateTablet for tablet e15163867cbf4cc2ab2d6da3e430b261 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a7bef71aae7647a990f6008892e44102]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:23.131700 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e15163867cbf4cc2ab2d6da3e430b261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:23.135641 28248 tablet_bootstrap.cc:492] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:23.139941 28248 tablet_bootstrap.cc:654] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:23.141968 28248 log.cc:824] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:23.146301 28251 tablet_bootstrap.cc:492] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:23.150787 28249 tablet_bootstrap.cc:492] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:23.154878 28251 tablet_bootstrap.cc:654] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:23.155057 28249 tablet_bootstrap.cc:654] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:23.155566 27237 tablet_service.cc:1467] Processing CreateTablet for tablet e15163867cbf4cc2ab2d6da3e430b261 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a7bef71aae7647a990f6008892e44102]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:23.156144 28249 log.cc:824] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:23.156935 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e15163867cbf4cc2ab2d6da3e430b261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:23.156903 28251 log.cc:824] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:23.157490 28248 tablet_bootstrap.cc:492] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:23.157856 28248 ts_tablet_manager.cc:1367] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.012s
I20240502 08:17:23.162364 27238 tablet_service.cc:1467] Processing CreateTablet for tablet d465bff0ba8c42458e7308bb85adfc65 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a7bef71aae7647a990f6008892e44102]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:23.162931 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d465bff0ba8c42458e7308bb85adfc65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:23.164731 28249 tablet_bootstrap.cc:492] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:23.165014 28249 ts_tablet_manager.cc:1367] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20240502 08:17:23.167740 28248 raft_consensus.cc:348] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:23.168862 28248 raft_consensus.cc:374] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:23.169009 28248 raft_consensus.cc:729] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:23.169826 28248 consensus_queue.cc:260] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:23.171340 28251 tablet_bootstrap.cc:492] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:23.171639 28251 ts_tablet_manager.cc:1367] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.007s
I20240502 08:17:23.174919 28249 raft_consensus.cc:348] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:23.175487 28249 raft_consensus.cc:374] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:23.175652 28249 raft_consensus.cc:729] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:23.176542 28249 consensus_queue.cc:260] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:23.177594 28255 tablet_bootstrap.cc:492] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:23.177917 28248 ts_tablet_manager.cc:1398] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.020s user 0.014s sys 0.003s
I20240502 08:17:23.179816 28248 tablet_bootstrap.cc:492] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:23.180335 28251 raft_consensus.cc:348] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:23.180728 28251 raft_consensus.cc:374] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:23.180817 28251 raft_consensus.cc:729] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:23.181298 28255 tablet_bootstrap.cc:654] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:23.181375 28248 tablet_bootstrap.cc:654] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:23.181515 28251 consensus_queue.cc:260] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:23.186852 28249 ts_tablet_manager.cc:1398] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.022s user 0.014s sys 0.000s
I20240502 08:17:23.188032 28249 tablet_bootstrap.cc:492] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:23.190362 28249 tablet_bootstrap.cc:654] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:23.190804 28255 log.cc:824] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:23.192987 28251 ts_tablet_manager.cc:1398] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.021s user 0.011s sys 0.003s
I20240502 08:17:23.193521 28251 tablet_bootstrap.cc:492] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:23.195206 28251 tablet_bootstrap.cc:654] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:23.195787 28248 tablet_bootstrap.cc:492] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:23.195971 28248 ts_tablet_manager.cc:1367] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20240502 08:17:23.198570 28248 raft_consensus.cc:348] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:23.198901 28248 raft_consensus.cc:374] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:23.199057 28248 raft_consensus.cc:729] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:23.199398 28248 consensus_queue.cc:260] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:23.200752 28255 tablet_bootstrap.cc:492] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:23.200760 28248 ts_tablet_manager.cc:1398] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:17:23.201001 28255 ts_tablet_manager.cc:1367] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.000s
I20240502 08:17:23.201386 28249 tablet_bootstrap.cc:492] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:23.201530 28249 ts_tablet_manager.cc:1367] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20240502 08:17:23.202468 28249 raft_consensus.cc:348] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:23.202731 28249 raft_consensus.cc:374] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:23.202883 28249 raft_consensus.cc:729] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:23.203194 28249 consensus_queue.cc:260] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:23.204530 28249 ts_tablet_manager.cc:1398] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:17:23.205549 28251 tablet_bootstrap.cc:492] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:23.205559 28257 raft_consensus.cc:482] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:23.205747 28251 ts_tablet_manager.cc:1367] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 08:17:23.205852 28257 raft_consensus.cc:504] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:23.207372 28251 raft_consensus.cc:348] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:23.207535 28251 raft_consensus.cc:374] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:23.207639 28251 raft_consensus.cc:729] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:23.207831 28251 consensus_queue.cc:260] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:23.210367 28257 leader_election.cc:288] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:23.210417 28255 raft_consensus.cc:348] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:23.210965 28255 raft_consensus.cc:374] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:23.211127 28255 raft_consensus.cc:729] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:23.212088 28255 consensus_queue.cc:260] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
W20240502 08:17:23.220571 27436 tablet.cc:2369] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:23.234846 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e15163867cbf4cc2ab2d6da3e430b261" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:23.235877 28251 ts_tablet_manager.cc:1398] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.030s user 0.004s sys 0.012s
I20240502 08:17:23.236707 28251 tablet_bootstrap.cc:492] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
W20240502 08:17:23.236644 27325 leader_election.cc:341] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:23.238471 28251 tablet_bootstrap.cc:654] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:23.238833 28255 ts_tablet_manager.cc:1398] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.038s user 0.021s sys 0.015s
W20240502 08:17:23.238934 27304 tablet.cc:2369] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:23.240070 28255 tablet_bootstrap.cc:492] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:23.241631 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e15163867cbf4cc2ab2d6da3e430b261" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:23.242233 27652 raft_consensus.cc:2454] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:23.242367 28251 tablet_bootstrap.cc:492] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:23.242527 28251 ts_tablet_manager.cc:1367] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240502 08:17:23.242822 28255 tablet_bootstrap.cc:654] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:23.243062 27326 leader_election.cc:302] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442; no voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60
I20240502 08:17:23.243250 28251 raft_consensus.cc:348] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:23.243486 28251 raft_consensus.cc:374] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:23.243568 28251 raft_consensus.cc:729] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:23.243782 28251 consensus_queue.cc:260] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:23.244040 28257 raft_consensus.cc:2781] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:23.244181 28257 raft_consensus.cc:482] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:23.244287 28251 ts_tablet_manager.cc:1398] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:23.244287 28257 raft_consensus.cc:3037] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:23.247234 28257 raft_consensus.cc:504] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:23.248245 28255 tablet_bootstrap.cc:492] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:23.248278 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e15163867cbf4cc2ab2d6da3e430b261" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:23.248453 28255 ts_tablet_manager.cc:1367] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240502 08:17:23.248445 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e15163867cbf4cc2ab2d6da3e430b261" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:23.248692 27652 raft_consensus.cc:3037] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:23.249066 28257 leader_election.cc:288] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
W20240502 08:17:23.249431 27325 leader_election.cc:341] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:23.249431 28255 raft_consensus.cc:348] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:23.249616 28255 raft_consensus.cc:374] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:23.249692 28255 raft_consensus.cc:729] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:23.249912 28255 consensus_queue.cc:260] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:23.250453 28255 ts_tablet_manager.cc:1398] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:23.251577 27652 raft_consensus.cc:2454] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:23.251922 27326 leader_election.cc:302] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442; no voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60
I20240502 08:17:23.252182 28257 raft_consensus.cc:2781] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:23.252329 28257 raft_consensus.cc:686] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:17:23.252709 28257 consensus_queue.cc:237] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:23.256914 27116 catalog_manager.cc:5484] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:23.304302 28265 raft_consensus.cc:482] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:23.304587 28265 raft_consensus.cc:504] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:23.306167 28265 leader_election.cc:288] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:23.314306 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d465bff0ba8c42458e7308bb85adfc65" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:23.314309 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d465bff0ba8c42458e7308bb85adfc65" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:23.314766 27390 raft_consensus.cc:2454] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:23.314838 27521 raft_consensus.cc:2454] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:23.315302 27193 leader_election.cc:302] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:23.315650 28265 raft_consensus.cc:2781] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:23.315809 28265 raft_consensus.cc:482] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:23.315908 28265 raft_consensus.cc:3037] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:23.317786 28265 raft_consensus.cc:504] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:23.318316 28265 leader_election.cc:288] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:23.318688 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d465bff0ba8c42458e7308bb85adfc65" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:23.318781 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d465bff0ba8c42458e7308bb85adfc65" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:23.318918 27521 raft_consensus.cc:3037] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:23.318974 27390 raft_consensus.cc:3037] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:23.321919 27521 raft_consensus.cc:2454] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:23.321919 27390 raft_consensus.cc:2454] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:23.322343 27194 leader_election.cc:302] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:23.322614 28265 raft_consensus.cc:2781] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:23.323030 28265 raft_consensus.cc:686] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:23.323457 28265 consensus_queue.cc:237] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:23.327600 27116 catalog_manager.cc:5484] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:23.368620 27698 tablet.cc:2369] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:23.390738 28254 raft_consensus.cc:482] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:23.390928 28254 raft_consensus.cc:504] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:23.392441 28254 leader_election.cc:288] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:23.400779 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f5a95b67d1c45578e323cc4ce8d4c14" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:23.400777 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f5a95b67d1c45578e323cc4ce8d4c14" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:23.401033 27390 raft_consensus.cc:2454] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:23.401036 27652 raft_consensus.cc:2454] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:23.401513 27455 leader_election.cc:302] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:23.401893 28254 raft_consensus.cc:2781] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:23.402046 28254 raft_consensus.cc:482] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:23.402135 28254 raft_consensus.cc:3037] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:23.404017 28254 raft_consensus.cc:504] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:23.404560 28254 leader_election.cc:288] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:23.404937 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f5a95b67d1c45578e323cc4ce8d4c14" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:23.404971 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f5a95b67d1c45578e323cc4ce8d4c14" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:23.405123 27652 raft_consensus.cc:3037] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:23.405185 27390 raft_consensus.cc:3037] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:23.406893 27652 raft_consensus.cc:2454] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:23.406930 27390 raft_consensus.cc:2454] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:23.407256 27456 leader_election.cc:302] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:23.407497 28254 raft_consensus.cc:2781] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:23.407907 28254 raft_consensus.cc:686] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:17:23.408371 28254 consensus_queue.cc:237] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:23.412196 27116 catalog_manager.cc:5484] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:23.418550 27567 tablet.cc:2369] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:23.529507 28265 raft_consensus.cc:482] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:23.529744 28265 raft_consensus.cc:504] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:23.530529 28265 leader_election.cc:288] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:23.531108 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e15163867cbf4cc2ab2d6da3e430b261" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:23.537060 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e15163867cbf4cc2ab2d6da3e430b261" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:23.537370 27652 raft_consensus.cc:2379] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in current term 1: Already voted for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in this term.
I20240502 08:17:23.537974 27194 leader_election.cc:302] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442
I20240502 08:17:23.538311 28265 raft_consensus.cc:3037] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:23.540252 28265 raft_consensus.cc:2726] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:23.720588 28265 consensus_queue.cc:1035] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:23.729089 28265 consensus_queue.cc:1035] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:23.758703 28257 consensus_queue.cc:1035] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:23.765285 28284 consensus_queue.cc:1035] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:23.813560 28277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:23.813863 28277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:23.820895 28277 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 08:17:23.930842 28254 consensus_queue.cc:1035] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:23.957684 28254 consensus_queue.cc:1035] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 323.124 ms
time per row: 0.161562 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:17:25.210413 27115 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:25.219738 27237 tablet_service.cc:1467] Processing CreateTablet for tablet af2f11471c064f48924191fc2607d31c (DEFAULT_TABLE table=rebalance_test_table_1 [id=33e20eb07c524d63b8e99e7957467d8b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:25.220115 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 15a333c65455473fbdc26d1ec137aebc (DEFAULT_TABLE table=rebalance_test_table_1 [id=33e20eb07c524d63b8e99e7957467d8b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:25.220283 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2f11471c064f48924191fc2607d31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:25.220504 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a333c65455473fbdc26d1ec137aebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:25.221618 27630 tablet_service.cc:1467] Processing CreateTablet for tablet af2f11471c064f48924191fc2607d31c (DEFAULT_TABLE table=rebalance_test_table_1 [id=33e20eb07c524d63b8e99e7957467d8b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:25.222038 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2f11471c064f48924191fc2607d31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:25.222060 27498 tablet_service.cc:1467] Processing CreateTablet for tablet f2b08a6b38f945b1b99f1d8ad1fd0ae0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=33e20eb07c524d63b8e99e7957467d8b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:25.222481 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b08a6b38f945b1b99f1d8ad1fd0ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:25.225864 27499 tablet_service.cc:1467] Processing CreateTablet for tablet af2f11471c064f48924191fc2607d31c (DEFAULT_TABLE table=rebalance_test_table_1 [id=33e20eb07c524d63b8e99e7957467d8b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:25.226251 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2f11471c064f48924191fc2607d31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:25.226363 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 15a333c65455473fbdc26d1ec137aebc (DEFAULT_TABLE table=rebalance_test_table_1 [id=33e20eb07c524d63b8e99e7957467d8b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:25.226675 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a333c65455473fbdc26d1ec137aebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:25.229525 27500 tablet_service.cc:1467] Processing CreateTablet for tablet 15a333c65455473fbdc26d1ec137aebc (DEFAULT_TABLE table=rebalance_test_table_1 [id=33e20eb07c524d63b8e99e7957467d8b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:25.229861 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a333c65455473fbdc26d1ec137aebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:25.235028 27370 tablet_service.cc:1467] Processing CreateTablet for tablet f2b08a6b38f945b1b99f1d8ad1fd0ae0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=33e20eb07c524d63b8e99e7957467d8b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:25.235415 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b08a6b38f945b1b99f1d8ad1fd0ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:25.237434 28356 tablet_bootstrap.cc:492] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:25.238801 27236 tablet_service.cc:1467] Processing CreateTablet for tablet f2b08a6b38f945b1b99f1d8ad1fd0ae0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=33e20eb07c524d63b8e99e7957467d8b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:25.239216 27236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b08a6b38f945b1b99f1d8ad1fd0ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:25.240484 28356 tablet_bootstrap.cc:654] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:25.249311 28356 tablet_bootstrap.cc:492] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:25.249563 28356 ts_tablet_manager.cc:1367] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240502 08:17:25.250614 28356 raft_consensus.cc:348] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:25.250989 28356 raft_consensus.cc:374] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:25.251169 28356 raft_consensus.cc:729] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:25.251523 28356 consensus_queue.cc:260] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:25.252208 28355 tablet_bootstrap.cc:492] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:25.254307 28355 tablet_bootstrap.cc:654] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:25.255107 28360 tablet_bootstrap.cc:492] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:25.255810 28357 tablet_bootstrap.cc:492] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:25.257552 28360 tablet_bootstrap.cc:654] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:25.257603 28357 tablet_bootstrap.cc:654] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:25.259194 28356 ts_tablet_manager.cc:1398] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20240502 08:17:25.259531 28356 tablet_bootstrap.cc:492] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:25.261276 28356 tablet_bootstrap.cc:654] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:25.263880 28355 tablet_bootstrap.cc:492] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:25.264104 28355 ts_tablet_manager.cc:1367] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240502 08:17:25.265064 28355 raft_consensus.cc:348] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:25.265364 28355 raft_consensus.cc:374] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:25.265512 28355 raft_consensus.cc:729] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:25.265841 28355 consensus_queue.cc:260] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:25.266688 28355 ts_tablet_manager.cc:1398] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:25.267086 28355 tablet_bootstrap.cc:492] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:25.267668 28356 tablet_bootstrap.cc:492] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:25.267920 28356 ts_tablet_manager.cc:1367] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20240502 08:17:25.268872 28355 tablet_bootstrap.cc:654] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:25.270656 28366 raft_consensus.cc:482] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:25.270888 28366 raft_consensus.cc:504] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:25.272202 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b08a6b38f945b1b99f1d8ad1fd0ae0" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:25.270437 28356 raft_consensus.cc:348] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:25.273098 28356 raft_consensus.cc:374] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 08:17:25.273037 27455 leader_election.cc:341] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:25.273416 28356 raft_consensus.cc:729] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:25.273721 28357 tablet_bootstrap.cc:492] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:25.273937 28357 ts_tablet_manager.cc:1367] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20240502 08:17:25.274984 28357 raft_consensus.cc:348] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:25.275334 28357 raft_consensus.cc:374] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:25.275501 28357 raft_consensus.cc:729] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:25.275859 28357 consensus_queue.cc:260] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:25.273835 28356 consensus_queue.cc:260] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:25.279866 28356 ts_tablet_manager.cc:1398] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 08:17:25.280791 28356 tablet_bootstrap.cc:492] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:25.283439 28360 tablet_bootstrap.cc:492] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:25.283643 28360 ts_tablet_manager.cc:1367] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.015s
I20240502 08:17:25.283615 28366 leader_election.cc:288] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:25.284230 28356 tablet_bootstrap.cc:654] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:25.284523 28360 raft_consensus.cc:348] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:25.284857 28360 raft_consensus.cc:374] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:25.285009 28360 raft_consensus.cc:729] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:25.285382 28360 consensus_queue.cc:260] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:25.286284 28357 ts_tablet_manager.cc:1398] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.012s user 0.000s sys 0.006s
I20240502 08:17:25.287012 28355 tablet_bootstrap.cc:492] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:25.287271 28355 ts_tablet_manager.cc:1367] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.004s
I20240502 08:17:25.287922 28360 ts_tablet_manager.cc:1398] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:17:25.288061 28355 raft_consensus.cc:348] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:25.288304 28355 raft_consensus.cc:374] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:25.288403 28355 raft_consensus.cc:729] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:25.288674 28355 consensus_queue.cc:260] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:25.289201 28357 tablet_bootstrap.cc:492] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:25.289407 28355 ts_tablet_manager.cc:1398] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:25.289693 28355 tablet_bootstrap.cc:492] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:25.291196 28355 tablet_bootstrap.cc:654] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:25.292783 28357 tablet_bootstrap.cc:654] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:25.293197 28355 tablet_bootstrap.cc:492] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:25.293324 28355 ts_tablet_manager.cc:1367] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:25.293498 28356 tablet_bootstrap.cc:492] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:25.293680 28356 ts_tablet_manager.cc:1367] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20240502 08:17:25.293973 28355 raft_consensus.cc:348] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:25.294193 28355 raft_consensus.cc:374] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:25.294289 28355 raft_consensus.cc:729] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:25.294535 28355 consensus_queue.cc:260] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:25.295049 28356 raft_consensus.cc:348] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:25.295310 28356 raft_consensus.cc:374] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:25.295454 28356 raft_consensus.cc:729] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:25.295529 28355 ts_tablet_manager.cc:1398] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:25.295809 28356 consensus_queue.cc:260] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:25.296538 28356 ts_tablet_manager.cc:1398] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:25.297122 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b08a6b38f945b1b99f1d8ad1fd0ae0" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:25.297363 27258 raft_consensus.cc:2454] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:25.297801 27455 leader_election.cc:302] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters: 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:17:25.298245 28366 raft_consensus.cc:2781] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:25.298352 28366 raft_consensus.cc:482] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:25.298435 28366 raft_consensus.cc:3037] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:25.298609 28357 tablet_bootstrap.cc:492] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:25.298751 28357 ts_tablet_manager.cc:1367] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240502 08:17:25.299520 28357 raft_consensus.cc:348] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:25.299729 28357 raft_consensus.cc:374] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:25.299801 28357 raft_consensus.cc:729] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:25.300045 28357 consensus_queue.cc:260] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:25.300426 28357 ts_tablet_manager.cc:1398] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:25.300595 28366 raft_consensus.cc:504] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:25.301054 28366 leader_election.cc:288] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:25.301505 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b08a6b38f945b1b99f1d8ad1fd0ae0" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:25.301589 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b08a6b38f945b1b99f1d8ad1fd0ae0" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:25.301708 27390 raft_consensus.cc:3037] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:25.301733 27258 raft_consensus.cc:3037] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:25.304477 27258 raft_consensus.cc:2454] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:25.304477 27390 raft_consensus.cc:2454] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:25.304828 27455 leader_election.cc:302] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:25.305075 28366 raft_consensus.cc:2781] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:25.305212 28366 raft_consensus.cc:686] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:17:25.305459 28366 consensus_queue.cc:237] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:25.307673 27107 catalog_manager.cc:5484] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:25.344745 28271 raft_consensus.cc:482] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:25.344900 28271 raft_consensus.cc:504] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:25.345546 28271 leader_election.cc:288] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:25.346019 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a333c65455473fbdc26d1ec137aebc" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:25.346089 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a333c65455473fbdc26d1ec137aebc" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:25.346213 27652 raft_consensus.cc:2454] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:25.346333 27521 raft_consensus.cc:2454] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:25.346613 27194 leader_election.cc:302] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:25.346840 28271 raft_consensus.cc:2781] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:25.346937 28271 raft_consensus.cc:482] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:25.347030 28271 raft_consensus.cc:3037] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:25.348829 28271 raft_consensus.cc:504] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:25.349292 28271 leader_election.cc:288] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:25.349720 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a333c65455473fbdc26d1ec137aebc" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:25.349749 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a333c65455473fbdc26d1ec137aebc" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:25.349908 27652 raft_consensus.cc:3037] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:25.349938 27521 raft_consensus.cc:3037] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:25.351756 28271 raft_consensus.cc:482] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:25.351910 28271 raft_consensus.cc:504] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:25.352438 28271 leader_election.cc:288] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:25.352686 27521 raft_consensus.cc:2454] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:25.352869 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2f11471c064f48924191fc2607d31c" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:25.352922 27652 raft_consensus.cc:2454] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:25.352917 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2f11471c064f48924191fc2607d31c" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:25.353081 27520 raft_consensus.cc:2454] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:25.353170 27651 raft_consensus.cc:2454] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:25.353341 27194 leader_election.cc:302] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:25.353586 28271 raft_consensus.cc:2781] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:25.353811 27194 leader_election.cc:302] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:25.353974 28271 raft_consensus.cc:686] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:25.354267 28375 raft_consensus.cc:2781] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:25.354405 28375 raft_consensus.cc:482] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:25.354591 28375 raft_consensus.cc:3037] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:25.354391 28271 consensus_queue.cc:237] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:25.357299 28375 raft_consensus.cc:504] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:25.357892 28375 leader_election.cc:288] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:25.357730 27107 catalog_manager.cc:5484] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:25.362710 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2f11471c064f48924191fc2607d31c" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:25.362757 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2f11471c064f48924191fc2607d31c" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:25.362865 27520 raft_consensus.cc:3037] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:25.362901 27651 raft_consensus.cc:3037] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:25.364589 27520 raft_consensus.cc:2454] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:25.364598 27651 raft_consensus.cc:2454] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:25.364908 27194 leader_election.cc:302] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:25.365183 28376 raft_consensus.cc:2781] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:25.365327 28376 raft_consensus.cc:686] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:25.365607 28376 consensus_queue.cc:237] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:25.368125 27107 catalog_manager.cc:5484] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:25.705989 28379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:25.706312 28379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:25.713330 28379 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 08:17:25.762569 28366 consensus_queue.cc:1035] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:25.763598 28376 consensus_queue.cc:1035] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:25.770310 28359 consensus_queue.cc:1035] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:25.786170 28375 consensus_queue.cc:1035] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:25.863564 28375 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:25.873114 28375 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 239.308 ms
time per row: 0.119654 ms
I20240502 08:17:26.101492 27109 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:26.109524 27370 tablet_service.cc:1467] Processing CreateTablet for tablet 9babdea7835440909cf3763ca9ffcb64 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1b92061ddf9546eb99771240ff39eac9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:26.110023 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9babdea7835440909cf3763ca9ffcb64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:26.110127 27368 tablet_service.cc:1467] Processing CreateTablet for tablet 82d6bd3211e34b2e85231b09b95e1587 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1b92061ddf9546eb99771240ff39eac9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:26.110098 27499 tablet_service.cc:1467] Processing CreateTablet for tablet 82d6bd3211e34b2e85231b09b95e1587 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1b92061ddf9546eb99771240ff39eac9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:26.110507 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d6bd3211e34b2e85231b09b95e1587. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:26.110486 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d6bd3211e34b2e85231b09b95e1587. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:26.110770 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 9babdea7835440909cf3763ca9ffcb64 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1b92061ddf9546eb99771240ff39eac9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:26.111168 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9babdea7835440909cf3763ca9ffcb64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:26.115096 27630 tablet_service.cc:1467] Processing CreateTablet for tablet 82d6bd3211e34b2e85231b09b95e1587 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1b92061ddf9546eb99771240ff39eac9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:26.115208 27629 tablet_service.cc:1467] Processing CreateTablet for tablet e095917962d4431c92c64190c7ddf55b (DEFAULT_TABLE table=rebalance_test_table_2 [id=1b92061ddf9546eb99771240ff39eac9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:26.115511 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d6bd3211e34b2e85231b09b95e1587. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:26.115895 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 9babdea7835440909cf3763ca9ffcb64 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1b92061ddf9546eb99771240ff39eac9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:26.116221 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9babdea7835440909cf3763ca9ffcb64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:26.117074 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e095917962d4431c92c64190c7ddf55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:26.123056 28438 tablet_bootstrap.cc:492] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:26.123025 27237 tablet_service.cc:1467] Processing CreateTablet for tablet e095917962d4431c92c64190c7ddf55b (DEFAULT_TABLE table=rebalance_test_table_2 [id=1b92061ddf9546eb99771240ff39eac9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:26.123401 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e095917962d4431c92c64190c7ddf55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:26.124805 28438 tablet_bootstrap.cc:654] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:26.128324 28437 tablet_bootstrap.cc:492] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:26.130847 28439 tablet_bootstrap.cc:492] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:26.132654 28437 tablet_bootstrap.cc:654] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:26.134553 28439 tablet_bootstrap.cc:654] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:26.134891 27370 tablet_service.cc:1467] Processing CreateTablet for tablet e095917962d4431c92c64190c7ddf55b (DEFAULT_TABLE table=rebalance_test_table_2 [id=1b92061ddf9546eb99771240ff39eac9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:26.135429 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e095917962d4431c92c64190c7ddf55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:26.141718 28441 tablet_bootstrap.cc:492] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:26.142047 28437 tablet_bootstrap.cc:492] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:26.142143 28438 tablet_bootstrap.cc:492] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:26.142182 28437 ts_tablet_manager.cc:1367] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.003s
I20240502 08:17:26.142308 28438 ts_tablet_manager.cc:1367] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.009s
I20240502 08:17:26.143054 28437 raft_consensus.cc:348] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:26.143249 28437 raft_consensus.cc:374] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:26.143218 28438 raft_consensus.cc:348] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:26.143461 28437 raft_consensus.cc:729] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:26.143472 28438 raft_consensus.cc:374] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:26.143564 28438 raft_consensus.cc:729] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:26.143805 28438 consensus_queue.cc:260] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:26.144670 28441 tablet_bootstrap.cc:654] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:26.144708 28439 tablet_bootstrap.cc:492] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:26.144853 28439 ts_tablet_manager.cc:1367] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20240502 08:17:26.145071 28438 ts_tablet_manager.cc:1398] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:26.144631 28437 consensus_queue.cc:260] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:26.145377 28438 tablet_bootstrap.cc:492] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:26.145677 28437 ts_tablet_manager.cc:1398] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:17:26.145730 28439 raft_consensus.cc:348] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:26.145968 28439 raft_consensus.cc:374] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:26.146106 28439 raft_consensus.cc:729] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:26.146476 28437 tablet_bootstrap.cc:492] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:26.146451 28439 consensus_queue.cc:260] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:26.147126 28439 ts_tablet_manager.cc:1398] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:26.147171 28438 tablet_bootstrap.cc:654] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:26.149446 28437 tablet_bootstrap.cc:654] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:26.157888 28441 tablet_bootstrap.cc:492] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:26.158092 28441 ts_tablet_manager.cc:1367] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20240502 08:17:26.158322 28437 tablet_bootstrap.cc:492] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:26.158509 28437 ts_tablet_manager.cc:1367] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240502 08:17:26.158957 28441 raft_consensus.cc:348] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:26.159260 28441 raft_consensus.cc:374] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:26.159407 28441 raft_consensus.cc:729] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:26.159358 28437 raft_consensus.cc:348] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:26.159605 28437 raft_consensus.cc:374] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:26.159714 28437 raft_consensus.cc:729] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:26.159770 28441 consensus_queue.cc:260] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:26.160143 28350 raft_consensus.cc:482] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:26.160096 28437 consensus_queue.cc:260] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:26.160322 28438 tablet_bootstrap.cc:492] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:26.160267 28350 raft_consensus.cc:504] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:26.160459 28438 ts_tablet_manager.cc:1367] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20240502 08:17:26.161113 28350 leader_election.cc:288] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:26.161072 28438 raft_consensus.cc:348] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:26.161245 28438 raft_consensus.cc:374] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:26.161305 28438 raft_consensus.cc:729] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:26.161527 28438 consensus_queue.cc:260] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:26.163818 28438 ts_tablet_manager.cc:1398] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:17:26.165858 28441 ts_tablet_manager.cc:1398] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:17:26.166271 28441 tablet_bootstrap.cc:492] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:26.165884 28438 tablet_bootstrap.cc:492] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:26.168391 28441 tablet_bootstrap.cc:654] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:26.169302 28438 tablet_bootstrap.cc:654] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:26.169744 28437 ts_tablet_manager.cc:1398] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240502 08:17:26.170473 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d6bd3211e34b2e85231b09b95e1587" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:26.170615 28437 tablet_bootstrap.cc:492] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:26.170728 27651 raft_consensus.cc:2454] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:26.171710 27326 leader_election.cc:302] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:26.172034 28437 tablet_bootstrap.cc:654] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:26.172376 28350 raft_consensus.cc:2781] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:26.172524 28350 raft_consensus.cc:482] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:26.172626 28350 raft_consensus.cc:3037] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:26.176345 28350 raft_consensus.cc:504] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:26.177088 28350 leader_election.cc:288] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:26.178289 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d6bd3211e34b2e85231b09b95e1587" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:26.178551 27652 raft_consensus.cc:3037] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:26.180079 28437 tablet_bootstrap.cc:492] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:26.180230 28437 ts_tablet_manager.cc:1367] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20240502 08:17:26.180562 28441 tablet_bootstrap.cc:492] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:26.180704 28441 ts_tablet_manager.cc:1367] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20240502 08:17:26.181391 28441 raft_consensus.cc:348] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:26.181473 28437 raft_consensus.cc:348] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:26.181598 28441 raft_consensus.cc:374] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:26.181653 28437 raft_consensus.cc:374] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:26.181665 28441 raft_consensus.cc:729] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:26.181738 28437 raft_consensus.cc:729] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:26.181967 28441 consensus_queue.cc:260] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:26.182009 28437 consensus_queue.cc:260] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:26.182559 28437 ts_tablet_manager.cc:1398] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:26.183336 27652 raft_consensus.cc:2454] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:26.183965 28438 tablet_bootstrap.cc:492] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:26.184150 28438 ts_tablet_manager.cc:1367] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20240502 08:17:26.184492 28441 ts_tablet_manager.cc:1398] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:26.184919 28438 raft_consensus.cc:348] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:26.185133 28438 raft_consensus.cc:374] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:26.185308 28438 raft_consensus.cc:729] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:26.185552 28438 consensus_queue.cc:260] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:26.186158 28438 ts_tablet_manager.cc:1398] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:26.186445 27326 leader_election.cc:302] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:26.186780 28350 raft_consensus.cc:2781] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:26.186975 28350 raft_consensus.cc:686] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:17:26.187279 28350 consensus_queue.cc:237] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:26.191139 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d6bd3211e34b2e85231b09b95e1587" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:26.191365 27521 raft_consensus.cc:2454] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:26.191879 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d6bd3211e34b2e85231b09b95e1587" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:26.192041 27521 raft_consensus.cc:3037] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:26.193712 27115 catalog_manager.cc:5484] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:26.194768 27521 raft_consensus.cc:2454] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:26.265771 28271 raft_consensus.cc:482] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:26.266024 28271 raft_consensus.cc:504] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:26.266978 28271 leader_election.cc:288] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:26.267410 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e095917962d4431c92c64190c7ddf55b" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:26.267449 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e095917962d4431c92c64190c7ddf55b" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:26.267609 27652 raft_consensus.cc:2454] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:26.267741 27390 raft_consensus.cc:2454] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:26.268014 27194 leader_election.cc:302] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:26.268342 28271 raft_consensus.cc:2781] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:26.268481 28271 raft_consensus.cc:482] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:26.268582 28271 raft_consensus.cc:3037] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:26.271489 28375 raft_consensus.cc:482] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:26.271646 28375 raft_consensus.cc:504] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:26.272370 28375 leader_election.cc:288] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:26.272619 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9babdea7835440909cf3763ca9ffcb64" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:26.272835 27390 raft_consensus.cc:2454] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:26.272935 28271 raft_consensus.cc:504] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:26.273435 27193 leader_election.cc:302] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:26.273684 28375 raft_consensus.cc:2781] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:26.273730 28271 leader_election.cc:288] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:26.273809 28375 raft_consensus.cc:482] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:26.273895 28375 raft_consensus.cc:3037] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:26.273923 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9babdea7835440909cf3763ca9ffcb64" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:26.274142 27652 raft_consensus.cc:2454] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:26.274641 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e095917962d4431c92c64190c7ddf55b" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:26.274880 27390 raft_consensus.cc:3037] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:26.275131 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e095917962d4431c92c64190c7ddf55b" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:26.275308 27652 raft_consensus.cc:3037] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:26.276607 28375 raft_consensus.cc:504] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:26.277165 28375 leader_election.cc:288] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:26.277923 27652 raft_consensus.cc:2454] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:26.278146 27390 raft_consensus.cc:2454] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:26.278371 27194 leader_election.cc:302] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:26.278631 28375 raft_consensus.cc:2781] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:26.278839 28375 raft_consensus.cc:686] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:26.279146 28375 consensus_queue.cc:237] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:26.279536 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9babdea7835440909cf3763ca9ffcb64" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:26.279707 27390 raft_consensus.cc:3037] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:26.281500 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9babdea7835440909cf3763ca9ffcb64" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:26.281708 27652 raft_consensus.cc:3037] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:26.282524 27115 catalog_manager.cc:5484] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:26.283408 27390 raft_consensus.cc:2454] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:26.283816 27193 leader_election.cc:302] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:26.284107 28375 raft_consensus.cc:2781] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:26.284168 27652 raft_consensus.cc:2454] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:26.284373 28375 raft_consensus.cc:686] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:26.284683 28375 consensus_queue.cc:237] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:26.287355 27115 catalog_manager.cc:5484] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } }
W20240502 08:17:26.642410 28455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:26.642809 28455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 08:17:26.646770 28350 consensus_queue.cc:1035] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:26.651540 28455 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:17:26.654417 28447 consensus_queue.cc:1035] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240502 08:17:26.683996 28375 consensus_queue.cc:1035] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:26.692466 28479 consensus_queue.cc:1035] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:26.731369 28479 consensus_queue.cc:1035] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:26.759492 28375 consensus_queue.cc:1035] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 224.143 ms
time per row: 0.112071 ms
I20240502 08:17:27.021741 27109 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:27.030454 27370 tablet_service.cc:1467] Processing CreateTablet for tablet 04a0a58eb52f41bf818e5102996d09e0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=cc946163c76846418f7f80018ed16ce5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:27.030927 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a0a58eb52f41bf818e5102996d09e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:27.030858 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 04a0a58eb52f41bf818e5102996d09e0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=cc946163c76846418f7f80018ed16ce5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:27.031311 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a0a58eb52f41bf818e5102996d09e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:27.031275 27368 tablet_service.cc:1467] Processing CreateTablet for tablet 7c233aeb1e58418ba7b01f48917fabad (DEFAULT_TABLE table=rebalance_test_table_3 [id=cc946163c76846418f7f80018ed16ce5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:27.031617 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c233aeb1e58418ba7b01f48917fabad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:27.032550 27237 tablet_service.cc:1467] Processing CreateTablet for tablet dd887cbb35c74e54a43d3dc6a2dde817 (DEFAULT_TABLE table=rebalance_test_table_3 [id=cc946163c76846418f7f80018ed16ce5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:27.032944 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd887cbb35c74e54a43d3dc6a2dde817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:27.035125 27629 tablet_service.cc:1467] Processing CreateTablet for tablet 7c233aeb1e58418ba7b01f48917fabad (DEFAULT_TABLE table=rebalance_test_table_3 [id=cc946163c76846418f7f80018ed16ce5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:27.035310 27630 tablet_service.cc:1467] Processing CreateTablet for tablet dd887cbb35c74e54a43d3dc6a2dde817 (DEFAULT_TABLE table=rebalance_test_table_3 [id=cc946163c76846418f7f80018ed16ce5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:27.035578 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c233aeb1e58418ba7b01f48917fabad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:27.036644 27499 tablet_service.cc:1467] Processing CreateTablet for tablet 04a0a58eb52f41bf818e5102996d09e0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=cc946163c76846418f7f80018ed16ce5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:27.037027 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a0a58eb52f41bf818e5102996d09e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:27.041229 27500 tablet_service.cc:1467] Processing CreateTablet for tablet 7c233aeb1e58418ba7b01f48917fabad (DEFAULT_TABLE table=rebalance_test_table_3 [id=cc946163c76846418f7f80018ed16ce5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:27.041684 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c233aeb1e58418ba7b01f48917fabad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:27.042411 27498 tablet_service.cc:1467] Processing CreateTablet for tablet dd887cbb35c74e54a43d3dc6a2dde817 (DEFAULT_TABLE table=rebalance_test_table_3 [id=cc946163c76846418f7f80018ed16ce5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:27.043081 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd887cbb35c74e54a43d3dc6a2dde817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:27.049578 28523 tablet_bootstrap.cc:492] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:27.050117 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd887cbb35c74e54a43d3dc6a2dde817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:27.051585 28525 tablet_bootstrap.cc:492] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:27.051718 28523 tablet_bootstrap.cc:654] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:27.053520 28525 tablet_bootstrap.cc:654] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:27.057334 28523 tablet_bootstrap.cc:492] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:27.057525 28523 ts_tablet_manager.cc:1367] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 08:17:27.059623 28523 raft_consensus.cc:348] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:27.059909 28523 raft_consensus.cc:374] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:27.060032 28523 raft_consensus.cc:729] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:27.060312 28523 consensus_queue.cc:260] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:27.061028 28523 ts_tablet_manager.cc:1398] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:27.061350 28523 tablet_bootstrap.cc:492] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:27.063360 28523 tablet_bootstrap.cc:654] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:27.070955 28523 tablet_bootstrap.cc:492] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:27.071163 28523 ts_tablet_manager.cc:1367] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 08:17:27.071416 28527 tablet_bootstrap.cc:492] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:27.072121 28523 raft_consensus.cc:348] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:27.072422 28523 raft_consensus.cc:374] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:27.072526 28523 raft_consensus.cc:729] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:27.072839 28523 consensus_queue.cc:260] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:27.072979 28527 tablet_bootstrap.cc:654] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:27.074110 28524 tablet_bootstrap.cc:492] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:27.075960 28524 tablet_bootstrap.cc:654] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:27.076149 28523 ts_tablet_manager.cc:1398] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:17:27.077622 28525 tablet_bootstrap.cc:492] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:27.077745 28525 ts_tablet_manager.cc:1367] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.008s
I20240502 08:17:27.077788 28527 tablet_bootstrap.cc:492] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:27.077911 28527 ts_tablet_manager.cc:1367] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 08:17:27.078425 28525 raft_consensus.cc:348] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:27.078579 28525 raft_consensus.cc:374] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:27.078547 28527 raft_consensus.cc:348] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:27.078657 28525 raft_consensus.cc:729] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:27.078724 28527 raft_consensus.cc:374] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:27.078855 28527 raft_consensus.cc:729] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:27.078891 28525 consensus_queue.cc:260] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:27.079044 28527 consensus_queue.cc:260] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:27.079535 28527 ts_tablet_manager.cc:1398] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:27.079769 28527 tablet_bootstrap.cc:492] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:27.080687 28525 ts_tablet_manager.cc:1398] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:27.080955 28525 tablet_bootstrap.cc:492] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:27.081385 28527 tablet_bootstrap.cc:654] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:27.082589 28525 tablet_bootstrap.cc:654] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:27.085014 28524 tablet_bootstrap.cc:492] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:27.085201 28524 ts_tablet_manager.cc:1367] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 08:17:27.085988 28524 raft_consensus.cc:348] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:27.086241 28524 raft_consensus.cc:374] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:27.086344 28524 raft_consensus.cc:729] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:27.086568 28524 consensus_queue.cc:260] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:27.086983 28527 tablet_bootstrap.cc:492] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:27.087122 28527 ts_tablet_manager.cc:1367] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.007s
I20240502 08:17:27.087469 28524 ts_tablet_manager.cc:1398] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:27.087702 28524 tablet_bootstrap.cc:492] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:27.087709 28527 raft_consensus.cc:348] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:27.087913 28527 raft_consensus.cc:374] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:27.087985 28527 raft_consensus.cc:729] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:27.088196 28527 consensus_queue.cc:260] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:27.088541 28512 raft_consensus.cc:482] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:27.088723 28527 ts_tablet_manager.cc:1398] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:27.088678 28512 raft_consensus.cc:504] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:27.088977 28527 tablet_bootstrap.cc:492] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:27.089366 28524 tablet_bootstrap.cc:654] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:27.089453 28512 leader_election.cc:288] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:27.089926 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd887cbb35c74e54a43d3dc6a2dde817" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:27.090255 27521 raft_consensus.cc:2454] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:27.090324 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd887cbb35c74e54a43d3dc6a2dde817" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:27.090942 28527 tablet_bootstrap.cc:654] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:27.091120 27194 leader_election.cc:302] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:27.091470 28512 raft_consensus.cc:2781] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240502 08:17:27.091619 27194 leader_election.cc:341] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:27.092700 28527 tablet_bootstrap.cc:492] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:27.092836 28527 ts_tablet_manager.cc:1367] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 08:17:27.091604 28512 raft_consensus.cc:482] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:27.093380 28512 raft_consensus.cc:3037] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:27.093559 28527 raft_consensus.cc:348] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:27.093735 28527 raft_consensus.cc:374] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:27.093835 28527 raft_consensus.cc:729] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:27.094070 28527 consensus_queue.cc:260] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:27.094492 28527 ts_tablet_manager.cc:1398] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:27.097229 28512 raft_consensus.cc:504] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:27.098531 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd887cbb35c74e54a43d3dc6a2dde817" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:27.098781 27521 raft_consensus.cc:3037] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:27.098951 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd887cbb35c74e54a43d3dc6a2dde817" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
W20240502 08:17:27.099542 27194 leader_election.cc:341] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:27.099689 28512 leader_election.cc:288] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:27.099730 28525 tablet_bootstrap.cc:492] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:27.099866 28525 ts_tablet_manager.cc:1367] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240502 08:17:27.100503 28525 raft_consensus.cc:348] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:27.100731 28525 raft_consensus.cc:374] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:27.100805 28525 raft_consensus.cc:729] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:27.101042 28525 consensus_queue.cc:260] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:27.101387 27521 raft_consensus.cc:2454] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:27.101567 28525 ts_tablet_manager.cc:1398] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:27.102432 27194 leader_election.cc:302] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters: bf8ac71b15f740a0a928911360773442
I20240502 08:17:27.102602 28524 tablet_bootstrap.cc:492] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:27.102741 28524 ts_tablet_manager.cc:1367] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.009s
I20240502 08:17:27.102826 28512 raft_consensus.cc:2781] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:27.103319 28512 raft_consensus.cc:686] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:27.103500 28524 raft_consensus.cc:348] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:27.103709 28524 raft_consensus.cc:374] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:27.103601 28512 consensus_queue.cc:237] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:27.103796 28524 raft_consensus.cc:729] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:27.104014 28524 consensus_queue.cc:260] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:27.104516 28524 ts_tablet_manager.cc:1398] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:27.106280 27110 catalog_manager.cc:5484] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:27.126448 28404 raft_consensus.cc:482] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:27.126588 28404 raft_consensus.cc:504] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:27.127154 28404 leader_election.cc:288] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:27.127585 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c233aeb1e58418ba7b01f48917fabad" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:27.127656 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c233aeb1e58418ba7b01f48917fabad" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:27.127799 27650 raft_consensus.cc:2454] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:27.127873 27389 raft_consensus.cc:2454] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:27.128168 27456 leader_election.cc:302] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:27.128453 28404 raft_consensus.cc:2781] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:27.128562 28404 raft_consensus.cc:482] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:27.128648 28404 raft_consensus.cc:3037] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:27.130596 28404 raft_consensus.cc:504] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:27.131431 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c233aeb1e58418ba7b01f48917fabad" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:27.131446 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c233aeb1e58418ba7b01f48917fabad" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:27.131615 27389 raft_consensus.cc:3037] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:27.131641 27650 raft_consensus.cc:3037] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:27.134315 27389 raft_consensus.cc:2454] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:27.134356 27650 raft_consensus.cc:2454] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:27.134554 28404 leader_election.cc:288] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:27.134780 27455 leader_election.cc:302] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:27.135083 28404 raft_consensus.cc:2781] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:27.135493 28404 raft_consensus.cc:686] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:17:27.135784 28404 consensus_queue.cc:237] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:27.138659 27110 catalog_manager.cc:5484] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:27.403683 28461 raft_consensus.cc:482] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:27.403853 28461 raft_consensus.cc:504] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:27.404450 28461 leader_election.cc:288] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:27.404896 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a0a58eb52f41bf818e5102996d09e0" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:27.404944 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a0a58eb52f41bf818e5102996d09e0" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:27.405074 27521 raft_consensus.cc:2454] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:27.405175 27258 raft_consensus.cc:2454] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:27.405448 27326 leader_election.cc:302] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:27.405702 28461 raft_consensus.cc:2781] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:27.405799 28461 raft_consensus.cc:482] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:27.405874 28461 raft_consensus.cc:3037] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:27.407694 28461 raft_consensus.cc:504] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:27.408186 28461 leader_election.cc:288] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:27.408509 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a0a58eb52f41bf818e5102996d09e0" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:27.408679 27521 raft_consensus.cc:3037] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:27.408738 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a0a58eb52f41bf818e5102996d09e0" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:27.408936 27258 raft_consensus.cc:3037] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:27.410593 27521 raft_consensus.cc:2454] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:27.410822 27258 raft_consensus.cc:2454] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:27.410902 27326 leader_election.cc:302] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:27.411201 28461 raft_consensus.cc:2781] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:27.411531 28461 raft_consensus.cc:686] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:17:27.411820 28461 consensus_queue.cc:237] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:27.414361 27110 catalog_manager.cc:5484] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:27.459074 28370 raft_consensus.cc:482] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:27.459280 28370 raft_consensus.cc:504] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:27.460830 28370 leader_election.cc:288] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:27.478920 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd887cbb35c74e54a43d3dc6a2dde817" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:27.479506 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd887cbb35c74e54a43d3dc6a2dde817" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:27.479776 27521 raft_consensus.cc:2379] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bf8ac71b15f740a0a928911360773442 in current term 1: Already voted for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in this term.
I20240502 08:17:27.480226 27587 leader_election.cc:302] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:17:27.480778 28370 raft_consensus.cc:3037] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:27.481710 28512 consensus_queue.cc:1035] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:27.483661 28370 raft_consensus.cc:2726] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:27.488190 28518 consensus_queue.cc:1035] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:27.653741 28404 consensus_queue.cc:1035] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:27.659250 28552 consensus_queue.cc:1035] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:27.774777 28541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:27.775099 28541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:27.782457 28541 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:17:27.803432 28461 consensus_queue.cc:1035] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:27.808269 28461 consensus_queue.cc:1035] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 218.624 ms
time per row: 0.109312 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:17:29.062352 27115 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:29.070858 27368 tablet_service.cc:1467] Processing CreateTablet for tablet 31edd3f3b1ec4b99b9bc7e16762e9911 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6da106d4ea7c4b0c887fce2b340247eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:29.071347 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31edd3f3b1ec4b99b9bc7e16762e9911. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:29.071632 27630 tablet_service.cc:1467] Processing CreateTablet for tablet 515abcc30e81445dae3103a40045691b (DEFAULT_TABLE table=rebalance_test_table_4 [id=6da106d4ea7c4b0c887fce2b340247eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:29.072053 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515abcc30e81445dae3103a40045691b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:29.072232 27370 tablet_service.cc:1467] Processing CreateTablet for tablet 515abcc30e81445dae3103a40045691b (DEFAULT_TABLE table=rebalance_test_table_4 [id=6da106d4ea7c4b0c887fce2b340247eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:29.072579 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515abcc30e81445dae3103a40045691b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:29.073009 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 515abcc30e81445dae3103a40045691b (DEFAULT_TABLE table=rebalance_test_table_4 [id=6da106d4ea7c4b0c887fce2b340247eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:29.073410 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515abcc30e81445dae3103a40045691b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:29.078485 27369 tablet_service.cc:1467] Processing CreateTablet for tablet f0332286514c49e2812434ec7043cb90 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6da106d4ea7c4b0c887fce2b340247eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:29.078606 27498 tablet_service.cc:1467] Processing CreateTablet for tablet 31edd3f3b1ec4b99b9bc7e16762e9911 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6da106d4ea7c4b0c887fce2b340247eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:29.078963 27369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0332286514c49e2812434ec7043cb90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:29.079072 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31edd3f3b1ec4b99b9bc7e16762e9911. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:29.082510 27237 tablet_service.cc:1467] Processing CreateTablet for tablet 31edd3f3b1ec4b99b9bc7e16762e9911 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6da106d4ea7c4b0c887fce2b340247eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:29.082860 27500 tablet_service.cc:1467] Processing CreateTablet for tablet f0332286514c49e2812434ec7043cb90 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6da106d4ea7c4b0c887fce2b340247eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:29.082994 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31edd3f3b1ec4b99b9bc7e16762e9911. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:29.083278 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0332286514c49e2812434ec7043cb90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:29.086416 28612 tablet_bootstrap.cc:492] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:29.086689 27629 tablet_service.cc:1467] Processing CreateTablet for tablet f0332286514c49e2812434ec7043cb90 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6da106d4ea7c4b0c887fce2b340247eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:29.087070 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0332286514c49e2812434ec7043cb90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:29.088771 28612 tablet_bootstrap.cc:654] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:29.099720 28615 tablet_bootstrap.cc:492] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:29.101842 28615 tablet_bootstrap.cc:654] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:29.103539 28613 tablet_bootstrap.cc:492] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:29.104646 28613 tablet_bootstrap.cc:654] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:29.106487 28616 tablet_bootstrap.cc:492] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:29.107488 28615 tablet_bootstrap.cc:492] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:29.107666 28615 ts_tablet_manager.cc:1367] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240502 08:17:29.107722 28613 tablet_bootstrap.cc:492] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:29.107751 28612 tablet_bootstrap.cc:492] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:29.107883 28612 ts_tablet_manager.cc:1367] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.006s
I20240502 08:17:29.107924 28613 ts_tablet_manager.cc:1367] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:17:29.107954 28616 tablet_bootstrap.cc:654] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:29.108616 28612 raft_consensus.cc:348] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:29.108616 28615 raft_consensus.cc:348] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:29.108888 28615 raft_consensus.cc:374] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:29.108886 28612 raft_consensus.cc:374] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:29.108721 28613 raft_consensus.cc:348] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:29.109026 28612 raft_consensus.cc:729] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:29.109032 28613 raft_consensus.cc:374] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:29.109143 28615 raft_consensus.cc:729] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:29.109138 28613 raft_consensus.cc:729] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:29.109267 28612 consensus_queue.cc:260] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:29.109539 28615 consensus_queue.cc:260] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:29.109849 28612 ts_tablet_manager.cc:1398] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:29.110255 28612 tablet_bootstrap.cc:492] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:29.109522 28613 consensus_queue.cc:260] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:29.112691 28612 tablet_bootstrap.cc:654] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:29.113518 28615 ts_tablet_manager.cc:1398] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 08:17:29.116634 28613 ts_tablet_manager.cc:1398] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240502 08:17:29.120085 28615 tablet_bootstrap.cc:492] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:29.120959 28612 tablet_bootstrap.cc:492] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:29.121160 28612 ts_tablet_manager.cc:1367] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240502 08:17:29.121544 28615 tablet_bootstrap.cc:654] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:29.122040 28612 raft_consensus.cc:348] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:29.122327 28612 raft_consensus.cc:374] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:29.122494 28612 raft_consensus.cc:729] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:29.122834 28612 consensus_queue.cc:260] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:29.123510 28612 ts_tablet_manager.cc:1398] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:29.123909 28612 tablet_bootstrap.cc:492] T 515abcc30e81445dae3103a40045691b P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:29.125449 28612 tablet_bootstrap.cc:654] T 515abcc30e81445dae3103a40045691b P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:29.130375 28615 tablet_bootstrap.cc:492] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:29.130563 28615 ts_tablet_manager.cc:1367] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240502 08:17:29.132838 28615 raft_consensus.cc:348] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:29.133109 28615 raft_consensus.cc:374] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:29.133206 28615 raft_consensus.cc:729] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:29.133525 28615 consensus_queue.cc:260] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:29.134289 28615 ts_tablet_manager.cc:1398] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:29.135425 28613 tablet_bootstrap.cc:492] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:29.137689 28613 tablet_bootstrap.cc:654] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:29.141983 28616 tablet_bootstrap.cc:492] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:29.142145 28616 ts_tablet_manager.cc:1367] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.025s
I20240502 08:17:29.142856 28616 raft_consensus.cc:348] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:29.143056 28616 raft_consensus.cc:374] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:29.143160 28616 raft_consensus.cc:729] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:29.143432 28616 consensus_queue.cc:260] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:29.143937 28613 tablet_bootstrap.cc:492] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:29.144076 28616 ts_tablet_manager.cc:1398] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:29.144105 28613 ts_tablet_manager.cc:1367] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 08:17:29.144376 28616 tablet_bootstrap.cc:492] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:29.144829 28613 raft_consensus.cc:348] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:29.144986 28612 tablet_bootstrap.cc:492] T 515abcc30e81445dae3103a40045691b P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:29.145049 28613 raft_consensus.cc:374] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:29.145125 28613 raft_consensus.cc:729] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:29.145135 28612 ts_tablet_manager.cc:1367] T 515abcc30e81445dae3103a40045691b P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.008s
I20240502 08:17:29.145383 28613 consensus_queue.cc:260] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:29.145900 28613 ts_tablet_manager.cc:1398] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:29.145931 28616 tablet_bootstrap.cc:654] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:29.145912 28612 raft_consensus.cc:348] T 515abcc30e81445dae3103a40045691b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:29.146133 28612 raft_consensus.cc:374] T 515abcc30e81445dae3103a40045691b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:29.146231 28612 raft_consensus.cc:729] T 515abcc30e81445dae3103a40045691b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:29.146446 28612 consensus_queue.cc:260] T 515abcc30e81445dae3103a40045691b P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:29.147112 28612 ts_tablet_manager.cc:1398] T 515abcc30e81445dae3103a40045691b P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:29.148696 28616 tablet_bootstrap.cc:492] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:29.148831 28616 ts_tablet_manager.cc:1367] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:17:29.149426 28616 raft_consensus.cc:348] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:29.149658 28616 raft_consensus.cc:374] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:29.149732 28616 raft_consensus.cc:729] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:29.149937 28616 consensus_queue.cc:260] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:29.150489 28616 ts_tablet_manager.cc:1398] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:29.150827 28620 raft_consensus.cc:482] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:29.150964 28620 raft_consensus.cc:504] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:29.151751 28620 leader_election.cc:288] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:29.153126 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515abcc30e81445dae3103a40045691b" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:29.153390 27257 raft_consensus.cc:2454] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:29.153949 27586 leader_election.cc:302] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:29.154304 28620 raft_consensus.cc:2781] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:29.154423 28620 raft_consensus.cc:482] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:29.154528 28620 raft_consensus.cc:3037] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:29.156775 28620 raft_consensus.cc:504] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:29.157296 28620 leader_election.cc:288] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:29.157696 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515abcc30e81445dae3103a40045691b" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:29.157852 27257 raft_consensus.cc:3037] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:29.158762 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515abcc30e81445dae3103a40045691b" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:29.158834 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515abcc30e81445dae3103a40045691b" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:29.159018 27390 raft_consensus.cc:2454] T 515abcc30e81445dae3103a40045691b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:29.160562 27257 raft_consensus.cc:2454] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:29.160913 27586 leader_election.cc:302] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters: 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:17:29.161177 28620 raft_consensus.cc:2781] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:29.161520 28620 raft_consensus.cc:686] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:17:29.161891 28620 consensus_queue.cc:237] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:29.165005 27107 catalog_manager.cc:5484] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:29.408300 28553 raft_consensus.cc:482] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:29.408465 28553 raft_consensus.cc:504] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:29.409114 28553 leader_election.cc:288] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:29.409552 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0332286514c49e2812434ec7043cb90" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:29.409646 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0332286514c49e2812434ec7043cb90" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:29.409790 27389 raft_consensus.cc:2454] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:29.409884 27650 raft_consensus.cc:2454] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:29.410223 27455 leader_election.cc:302] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:29.410537 28553 raft_consensus.cc:2781] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:29.410634 28553 raft_consensus.cc:482] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:29.410718 28553 raft_consensus.cc:3037] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:29.412582 28553 raft_consensus.cc:504] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:29.413058 28553 leader_election.cc:288] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:29.413476 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0332286514c49e2812434ec7043cb90" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:29.413481 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0332286514c49e2812434ec7043cb90" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:29.413679 27389 raft_consensus.cc:3037] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:29.413689 27650 raft_consensus.cc:3037] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:29.416427 27650 raft_consensus.cc:2454] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:29.416450 27389 raft_consensus.cc:2454] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:29.416843 27456 leader_election.cc:302] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:29.417129 28553 raft_consensus.cc:2781] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:29.417423 28553 raft_consensus.cc:686] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:17:29.417690 28553 consensus_queue.cc:237] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:29.420076 27107 catalog_manager.cc:5484] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:29.451326 28518 raft_consensus.cc:482] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:29.451532 28518 raft_consensus.cc:504] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:29.452136 28518 leader_election.cc:288] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:29.452476 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31edd3f3b1ec4b99b9bc7e16762e9911" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:29.452553 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31edd3f3b1ec4b99b9bc7e16762e9911" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:29.452718 27389 raft_consensus.cc:2454] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:29.452759 27521 raft_consensus.cc:2454] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:29.453085 27193 leader_election.cc:302] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:29.453364 28518 raft_consensus.cc:2781] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:29.453462 28518 raft_consensus.cc:482] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:29.453567 28518 raft_consensus.cc:3037] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:29.455796 28518 raft_consensus.cc:504] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:29.456313 28518 leader_election.cc:288] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:29.456640 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31edd3f3b1ec4b99b9bc7e16762e9911" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:29.456692 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31edd3f3b1ec4b99b9bc7e16762e9911" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:29.456845 27389 raft_consensus.cc:3037] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:29.456861 27521 raft_consensus.cc:3037] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:29.459568 27521 raft_consensus.cc:2454] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:29.459565 27389 raft_consensus.cc:2454] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:29.459937 27193 leader_election.cc:302] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:29.460181 28518 raft_consensus.cc:2781] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:29.460318 28518 raft_consensus.cc:686] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:29.460542 28518 consensus_queue.cc:237] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:29.462914 27107 catalog_manager.cc:5484] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:29.647495 27389 raft_consensus.cc:3037] T 515abcc30e81445dae3103a40045691b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:29.651039 28620 consensus_queue.cc:1035] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:29.658813 28620 consensus_queue.cc:1035] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:29.818734 28553 consensus_queue.cc:1035] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:29.825572 28553 consensus_queue.cc:1035] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:29.871913 28633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:29.872220 28633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:29.882436 28633 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 08:17:30.043731 27390 raft_consensus.cc:1261] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:30.044364 28665 consensus_queue.cc:1035] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:30.064339 28665 consensus_queue.cc:1035] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:30.110457 28640 mvcc.cc:205] Tried to move back new op lower bound from 7023156633772081152 to 7023156631392047104. Current Snapshot: MvccSnapshot[applied={T|T < 7023156633772081152}]
I20240502 08:17:30.115448 28676 mvcc.cc:205] Tried to move back new op lower bound from 7023156633944330240 to 7023156631392047104. Current Snapshot: MvccSnapshot[applied={T|T < 7023156633772081152}]
I20240502 08:17:30.156988 28679 mvcc.cc:205] Tried to move back new op lower bound from 7023156633772081152 to 7023156631392047104. Current Snapshot: MvccSnapshot[applied={T|T < 7023156633772081152}]
INSERT report
rows total: 2000
time total: 259.493 ms
time per row: 0.129747 ms
I20240502 08:17:30.290167 27115 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:30.298094 27237 tablet_service.cc:1467] Processing CreateTablet for tablet bd640e48563f458cb2d56c94ed1e0347 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5c5af31071894d4f803992d5173e0429]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:30.298554 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd640e48563f458cb2d56c94ed1e0347. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:30.298635 27630 tablet_service.cc:1467] Processing CreateTablet for tablet bd640e48563f458cb2d56c94ed1e0347 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5c5af31071894d4f803992d5173e0429]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:30.299059 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd640e48563f458cb2d56c94ed1e0347. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:30.299798 27238 tablet_service.cc:1467] Processing CreateTablet for tablet c2776f857a5f47e9b8022639dc54757d (DEFAULT_TABLE table=rebalance_test_table_5 [id=5c5af31071894d4f803992d5173e0429]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:30.300035 27236 tablet_service.cc:1467] Processing CreateTablet for tablet 0bea60d096f9408c9f4a857f8824d6a2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5c5af31071894d4f803992d5173e0429]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:30.300168 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2776f857a5f47e9b8022639dc54757d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:30.300380 27236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bea60d096f9408c9f4a857f8824d6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:30.301774 27370 tablet_service.cc:1467] Processing CreateTablet for tablet 0bea60d096f9408c9f4a857f8824d6a2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5c5af31071894d4f803992d5173e0429]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:30.302163 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bea60d096f9408c9f4a857f8824d6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:30.305717 27500 tablet_service.cc:1467] Processing CreateTablet for tablet bd640e48563f458cb2d56c94ed1e0347 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5c5af31071894d4f803992d5173e0429]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:30.306140 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd640e48563f458cb2d56c94ed1e0347. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:30.306066 27629 tablet_service.cc:1467] Processing CreateTablet for tablet c2776f857a5f47e9b8022639dc54757d (DEFAULT_TABLE table=rebalance_test_table_5 [id=5c5af31071894d4f803992d5173e0429]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:30.306560 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2776f857a5f47e9b8022639dc54757d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:30.309535 27498 tablet_service.cc:1467] Processing CreateTablet for tablet c2776f857a5f47e9b8022639dc54757d (DEFAULT_TABLE table=rebalance_test_table_5 [id=5c5af31071894d4f803992d5173e0429]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:30.309875 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2776f857a5f47e9b8022639dc54757d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:30.314455 27499 tablet_service.cc:1467] Processing CreateTablet for tablet 0bea60d096f9408c9f4a857f8824d6a2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5c5af31071894d4f803992d5173e0429]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:30.314839 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bea60d096f9408c9f4a857f8824d6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:30.315531 28706 tablet_bootstrap.cc:492] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:30.317234 28706 tablet_bootstrap.cc:654] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:30.324121 28704 tablet_bootstrap.cc:492] T bd640e48563f458cb2d56c94ed1e0347 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:30.326224 28704 tablet_bootstrap.cc:654] T bd640e48563f458cb2d56c94ed1e0347 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:30.329815 28706 tablet_bootstrap.cc:492] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:30.330088 28706 ts_tablet_manager.cc:1367] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20240502 08:17:30.330529 28704 tablet_bootstrap.cc:492] T bd640e48563f458cb2d56c94ed1e0347 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:30.331099 28706 raft_consensus.cc:348] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:30.331521 28706 raft_consensus.cc:374] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:30.331539 28705 tablet_bootstrap.cc:492] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:30.331657 28706 raft_consensus.cc:729] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:30.331977 28709 tablet_bootstrap.cc:492] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:30.331907 28706 consensus_queue.cc:260] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:30.332024 28704 ts_tablet_manager.cc:1367] T bd640e48563f458cb2d56c94ed1e0347 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20240502 08:17:30.332754 28704 raft_consensus.cc:348] T bd640e48563f458cb2d56c94ed1e0347 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:30.332988 28704 raft_consensus.cc:374] T bd640e48563f458cb2d56c94ed1e0347 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:30.333145 28704 raft_consensus.cc:729] T bd640e48563f458cb2d56c94ed1e0347 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:30.333599 28705 tablet_bootstrap.cc:654] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:30.333470 28704 consensus_queue.cc:260] T bd640e48563f458cb2d56c94ed1e0347 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:30.333631 28709 tablet_bootstrap.cc:654] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:30.334064 28704 ts_tablet_manager.cc:1398] T bd640e48563f458cb2d56c94ed1e0347 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:30.334440 28704 tablet_bootstrap.cc:492] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:30.336014 28704 tablet_bootstrap.cc:654] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:30.336321 28706 ts_tablet_manager.cc:1398] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:17:30.343724 28705 tablet_bootstrap.cc:492] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:30.343851 28705 ts_tablet_manager.cc:1367] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240502 08:17:30.343894 28704 tablet_bootstrap.cc:492] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:30.344043 28704 ts_tablet_manager.cc:1367] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240502 08:17:30.344439 28709 tablet_bootstrap.cc:492] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:30.344574 28709 ts_tablet_manager.cc:1367] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20240502 08:17:30.344504 28705 raft_consensus.cc:348] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:30.344712 28705 raft_consensus.cc:374] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:30.344813 28705 raft_consensus.cc:729] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:30.345048 28705 consensus_queue.cc:260] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:30.345149 28704 raft_consensus.cc:348] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:30.345378 28709 raft_consensus.cc:348] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:30.345503 28704 raft_consensus.cc:374] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:30.345571 28709 raft_consensus.cc:374] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:30.345574 28704 raft_consensus.cc:729] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:30.345602 28705 ts_tablet_manager.cc:1398] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:30.345654 28709 raft_consensus.cc:729] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:30.345808 28704 consensus_queue.cc:260] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:30.346340 28704 ts_tablet_manager.cc:1398] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:30.346035 28705 tablet_bootstrap.cc:492] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:30.345863 28709 consensus_queue.cc:260] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:30.347131 28709 ts_tablet_manager.cc:1398] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:30.347959 28705 tablet_bootstrap.cc:654] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:30.348889 28709 tablet_bootstrap.cc:492] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:30.350891 28705 tablet_bootstrap.cc:492] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:30.351039 28705 ts_tablet_manager.cc:1367] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 08:17:30.351239 28709 tablet_bootstrap.cc:654] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:30.351748 28705 raft_consensus.cc:348] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:30.351961 28705 raft_consensus.cc:374] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:30.352048 28705 raft_consensus.cc:729] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:30.352358 28705 consensus_queue.cc:260] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:30.352720 28705 ts_tablet_manager.cc:1398] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:30.353021 28705 tablet_bootstrap.cc:492] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:30.353353 28709 tablet_bootstrap.cc:492] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:30.353492 28709 ts_tablet_manager.cc:1367] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240502 08:17:30.354058 28709 raft_consensus.cc:348] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:30.354261 28709 raft_consensus.cc:374] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:30.354341 28709 raft_consensus.cc:729] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:30.354472 28705 tablet_bootstrap.cc:654] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:30.354574 28709 consensus_queue.cc:260] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:30.355082 28709 ts_tablet_manager.cc:1398] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:30.355346 28709 tablet_bootstrap.cc:492] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:30.356734 28709 tablet_bootstrap.cc:654] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:30.356938 28705 tablet_bootstrap.cc:492] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:30.357056 28705 ts_tablet_manager.cc:1367] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:30.357620 28705 raft_consensus.cc:348] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:30.357774 28705 raft_consensus.cc:374] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:30.357853 28705 raft_consensus.cc:729] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:30.358062 28705 consensus_queue.cc:260] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:30.358572 28705 ts_tablet_manager.cc:1398] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:17:30.358659 28709 tablet_bootstrap.cc:492] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:30.358816 28709 ts_tablet_manager.cc:1367] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:30.359565 28709 raft_consensus.cc:348] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:30.359704 28709 raft_consensus.cc:374] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:30.359797 28709 raft_consensus.cc:729] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:30.360004 28709 consensus_queue.cc:260] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:30.360333 28709 ts_tablet_manager.cc:1398] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:17:30.401409 28553 raft_consensus.cc:482] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:30.401650 28553 raft_consensus.cc:504] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:30.402194 28553 leader_election.cc:288] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:30.402684 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bea60d096f9408c9f4a857f8824d6a2" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:30.402704 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bea60d096f9408c9f4a857f8824d6a2" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:30.402892 27390 raft_consensus.cc:2454] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:30.402907 27257 raft_consensus.cc:2454] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:30.403324 27455 leader_election.cc:302] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:30.403578 28553 raft_consensus.cc:2781] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:30.403658 28553 raft_consensus.cc:482] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:30.403755 28553 raft_consensus.cc:3037] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:30.405567 28553 raft_consensus.cc:504] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:30.406095 28553 leader_election.cc:288] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:30.406517 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bea60d096f9408c9f4a857f8824d6a2" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:30.406558 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bea60d096f9408c9f4a857f8824d6a2" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:30.406680 27390 raft_consensus.cc:3037] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:30.406733 27257 raft_consensus.cc:3037] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:30.408576 27257 raft_consensus.cc:2454] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:30.408638 27390 raft_consensus.cc:2454] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:30.408919 27455 leader_election.cc:302] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:30.409143 28553 raft_consensus.cc:2781] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:30.409298 28553 raft_consensus.cc:686] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:17:30.409547 28553 consensus_queue.cc:237] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:30.412039 27107 catalog_manager.cc:5484] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:30.437505 28553 raft_consensus.cc:482] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:30.437661 28553 raft_consensus.cc:504] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:30.438217 28553 leader_election.cc:288] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:30.438695 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2776f857a5f47e9b8022639dc54757d" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:30.438680 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2776f857a5f47e9b8022639dc54757d" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:30.438941 27257 raft_consensus.cc:2454] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:30.438941 27650 raft_consensus.cc:2454] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:30.439328 27455 leader_election.cc:302] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:30.439602 28681 raft_consensus.cc:2781] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:30.439703 28681 raft_consensus.cc:482] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:30.439818 28681 raft_consensus.cc:3037] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:30.459957 28681 raft_consensus.cc:504] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:30.460527 28681 leader_election.cc:288] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:30.461081 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2776f857a5f47e9b8022639dc54757d" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:30.461073 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2776f857a5f47e9b8022639dc54757d" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:30.461299 27257 raft_consensus.cc:3037] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:30.461299 27650 raft_consensus.cc:3037] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:30.464043 27257 raft_consensus.cc:2454] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:30.464043 27650 raft_consensus.cc:2454] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:30.464493 27455 leader_election.cc:302] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:30.464758 28681 raft_consensus.cc:2781] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:30.464936 28681 raft_consensus.cc:686] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:17:30.465216 28681 consensus_queue.cc:237] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:30.467872 27107 catalog_manager.cc:5484] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:30.535020 28518 raft_consensus.cc:482] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:30.535171 28518 raft_consensus.cc:504] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:30.535719 28518 leader_election.cc:288] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:30.536185 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd640e48563f458cb2d56c94ed1e0347" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:30.536206 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd640e48563f458cb2d56c94ed1e0347" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:30.536396 27521 raft_consensus.cc:2454] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:30.536417 27650 raft_consensus.cc:2454] T bd640e48563f458cb2d56c94ed1e0347 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:30.536748 27194 leader_election.cc:302] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:30.536973 28518 raft_consensus.cc:2781] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:30.537079 28518 raft_consensus.cc:482] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:30.537170 28518 raft_consensus.cc:3037] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:30.538985 28518 raft_consensus.cc:504] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:30.539448 28518 leader_election.cc:288] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:30.539806 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd640e48563f458cb2d56c94ed1e0347" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:30.539851 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd640e48563f458cb2d56c94ed1e0347" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:30.539990 27521 raft_consensus.cc:3037] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:30.540062 27650 raft_consensus.cc:3037] T bd640e48563f458cb2d56c94ed1e0347 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:30.541800 27650 raft_consensus.cc:2454] T bd640e48563f458cb2d56c94ed1e0347 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:30.541927 27521 raft_consensus.cc:2454] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:30.542130 27194 leader_election.cc:302] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:30.542367 28518 raft_consensus.cc:2781] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:30.542527 28518 raft_consensus.cc:686] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:30.542804 28518 consensus_queue.cc:237] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:30.545259 27107 catalog_manager.cc:5484] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:30.798792 28681 consensus_queue.cc:1035] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:30.809089 28553 consensus_queue.cc:1035] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:30.883123 28720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:30.883395 28720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:30.890723 28720 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 08:17:30.922464 28518 consensus_queue.cc:1035] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:30.928493 28518 consensus_queue.cc:1035] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:30.929304 28681 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:30.942567 28723 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 295.624 ms
time per row: 0.147812 ms
-----------------------------------------------------
Suppressions used:
count bytes template
14 3624 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:17:32.279335 27107 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:32.289000 27370 tablet_service.cc:1467] Processing CreateTablet for tablet 0866b26202c14efca9a753fa1d2e8071 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9db3ba2ceb8c4fc8835b074aa27409c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:32.289173 27237 tablet_service.cc:1467] Processing CreateTablet for tablet 0866b26202c14efca9a753fa1d2e8071 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9db3ba2ceb8c4fc8835b074aa27409c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:32.289403 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0866b26202c14efca9a753fa1d2e8071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:32.289605 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0866b26202c14efca9a753fa1d2e8071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:32.290163 27630 tablet_service.cc:1467] Processing CreateTablet for tablet 0866b26202c14efca9a753fa1d2e8071 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9db3ba2ceb8c4fc8835b074aa27409c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:32.290556 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0866b26202c14efca9a753fa1d2e8071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:32.290781 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 0662521f50104435aedf647a56f57c33 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9db3ba2ceb8c4fc8835b074aa27409c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:32.291141 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0662521f50104435aedf647a56f57c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:32.294879 27629 tablet_service.cc:1467] Processing CreateTablet for tablet 72eb1c4f44934edfb49a3d6a39618df4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9db3ba2ceb8c4fc8835b074aa27409c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:32.294878 27368 tablet_service.cc:1467] Processing CreateTablet for tablet 0662521f50104435aedf647a56f57c33 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9db3ba2ceb8c4fc8835b074aa27409c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:32.295274 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72eb1c4f44934edfb49a3d6a39618df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:32.295275 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0662521f50104435aedf647a56f57c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:32.298465 27369 tablet_service.cc:1467] Processing CreateTablet for tablet 72eb1c4f44934edfb49a3d6a39618df4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9db3ba2ceb8c4fc8835b074aa27409c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:32.298956 27369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72eb1c4f44934edfb49a3d6a39618df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:32.302871 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 0662521f50104435aedf647a56f57c33 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9db3ba2ceb8c4fc8835b074aa27409c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:32.303198 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0662521f50104435aedf647a56f57c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:32.303404 27500 tablet_service.cc:1467] Processing CreateTablet for tablet 72eb1c4f44934edfb49a3d6a39618df4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9db3ba2ceb8c4fc8835b074aa27409c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:32.303786 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72eb1c4f44934edfb49a3d6a39618df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:32.307583 28789 tablet_bootstrap.cc:492] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:32.310402 28789 tablet_bootstrap.cc:654] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:32.311791 28787 tablet_bootstrap.cc:492] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:32.311834 28788 tablet_bootstrap.cc:492] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:32.313413 28788 tablet_bootstrap.cc:654] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:32.313580 28787 tablet_bootstrap.cc:654] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:32.325304 28789 tablet_bootstrap.cc:492] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:32.325557 28789 ts_tablet_manager.cc:1367] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20240502 08:17:32.326737 28789 raft_consensus.cc:348] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:32.327116 28789 raft_consensus.cc:374] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:32.327282 28789 raft_consensus.cc:729] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:32.327610 28789 consensus_queue.cc:260] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:32.328513 28789 ts_tablet_manager.cc:1398] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:17:32.328881 28789 tablet_bootstrap.cc:492] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:32.330435 28789 tablet_bootstrap.cc:654] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:32.334093 28793 tablet_bootstrap.cc:492] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:32.336858 28793 tablet_bootstrap.cc:654] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:32.339324 28787 tablet_bootstrap.cc:492] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:32.339485 28789 tablet_bootstrap.cc:492] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:32.339522 28787 ts_tablet_manager.cc:1367] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.028s user 0.017s sys 0.007s
I20240502 08:17:32.339622 28788 tablet_bootstrap.cc:492] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:32.339685 28789 ts_tablet_manager.cc:1367] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 08:17:32.339843 28788 ts_tablet_manager.cc:1367] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.015s
I20240502 08:17:32.339860 28793 tablet_bootstrap.cc:492] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:32.339987 28793 ts_tablet_manager.cc:1367] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:17:32.340353 28787 raft_consensus.cc:348] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:32.340476 28789 raft_consensus.cc:348] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:32.340631 28787 raft_consensus.cc:374] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:32.340698 28789 raft_consensus.cc:374] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:32.340660 28788 raft_consensus.cc:348] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:32.340792 28789 raft_consensus.cc:729] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:32.340705 28793 raft_consensus.cc:348] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:32.340807 28787 raft_consensus.cc:729] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:32.340881 28793 raft_consensus.cc:374] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:32.340934 28788 raft_consensus.cc:374] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:32.340996 28793 raft_consensus.cc:729] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:32.341084 28788 raft_consensus.cc:729] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:32.341042 28787 consensus_queue.cc:260] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:32.341056 28789 consensus_queue.cc:260] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:32.341267 28793 consensus_queue.cc:260] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:32.341387 28788 consensus_queue.cc:260] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:32.341956 28793 ts_tablet_manager.cc:1398] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:32.342075 28788 ts_tablet_manager.cc:1398] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:32.342936 28787 ts_tablet_manager.cc:1398] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:32.343199 28787 tablet_bootstrap.cc:492] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:32.343737 28789 ts_tablet_manager.cc:1398] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:17:32.344041 28789 tablet_bootstrap.cc:492] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:32.344704 28787 tablet_bootstrap.cc:654] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:32.343106 28788 tablet_bootstrap.cc:492] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:32.349260 28789 tablet_bootstrap.cc:654] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:32.350878 28788 tablet_bootstrap.cc:654] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:32.354460 28787 tablet_bootstrap.cc:492] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:32.354624 28787 ts_tablet_manager.cc:1367] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20240502 08:17:32.354744 28788 tablet_bootstrap.cc:492] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:32.354911 28788 ts_tablet_manager.cc:1367] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240502 08:17:32.355033 28789 tablet_bootstrap.cc:492] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:32.355171 28789 ts_tablet_manager.cc:1367] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20240502 08:17:32.355455 28787 raft_consensus.cc:348] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:32.355572 28788 raft_consensus.cc:348] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:32.355672 28787 raft_consensus.cc:374] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:32.355751 28787 raft_consensus.cc:729] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:32.355780 28788 raft_consensus.cc:374] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:32.355886 28788 raft_consensus.cc:729] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:32.355926 28789 raft_consensus.cc:348] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:32.356122 28789 raft_consensus.cc:374] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:32.355995 28787 consensus_queue.cc:260] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:32.356209 28789 raft_consensus.cc:729] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:32.356128 28788 consensus_queue.cc:260] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:32.356688 28788 ts_tablet_manager.cc:1398] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:32.356638 28789 consensus_queue.cc:260] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:32.357165 28789 ts_tablet_manager.cc:1398] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:32.357190 28787 ts_tablet_manager.cc:1398] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:32.357419 28787 tablet_bootstrap.cc:492] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:32.358744 28787 tablet_bootstrap.cc:654] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:32.366896 28787 tablet_bootstrap.cc:492] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:32.367015 28787 ts_tablet_manager.cc:1367] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20240502 08:17:32.367553 28787 raft_consensus.cc:348] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:32.367709 28787 raft_consensus.cc:374] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:32.367805 28787 raft_consensus.cc:729] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:32.367980 28787 consensus_queue.cc:260] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:32.368467 28787 ts_tablet_manager.cc:1398] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:32.397956 28782 raft_consensus.cc:482] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:32.398155 28782 raft_consensus.cc:504] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:32.398810 28782 leader_election.cc:288] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:32.399223 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0662521f50104435aedf647a56f57c33" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:32.399276 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0662521f50104435aedf647a56f57c33" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:32.399436 27651 raft_consensus.cc:2454] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:32.399533 27389 raft_consensus.cc:2454] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:32.399912 27194 leader_election.cc:302] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:32.400163 28782 raft_consensus.cc:2781] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:32.400275 28782 raft_consensus.cc:482] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:32.400364 28782 raft_consensus.cc:3037] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:32.402266 28782 raft_consensus.cc:504] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:32.402823 28782 leader_election.cc:288] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:32.403164 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0662521f50104435aedf647a56f57c33" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:32.403348 27651 raft_consensus.cc:3037] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:32.403303 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0662521f50104435aedf647a56f57c33" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:32.403513 27389 raft_consensus.cc:3037] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:32.405151 27651 raft_consensus.cc:2454] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:32.405388 27389 raft_consensus.cc:2454] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:32.405697 27193 leader_election.cc:302] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:32.405987 28782 raft_consensus.cc:2781] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:32.406183 28782 raft_consensus.cc:686] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:32.406473 28782 consensus_queue.cc:237] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:32.409571 27115 catalog_manager.cc:5484] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:32.440292 28628 raft_consensus.cc:482] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:32.440435 28628 raft_consensus.cc:504] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:32.441000 28628 leader_election.cc:288] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:32.441363 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72eb1c4f44934edfb49a3d6a39618df4" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:32.441408 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72eb1c4f44934edfb49a3d6a39618df4" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:32.441540 27389 raft_consensus.cc:2454] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:32.441625 27521 raft_consensus.cc:2454] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:32.441916 27586 leader_election.cc:302] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:32.442180 28628 raft_consensus.cc:2781] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:32.442281 28628 raft_consensus.cc:482] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:32.442337 28628 raft_consensus.cc:3037] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:32.444058 28628 raft_consensus.cc:504] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:32.444501 28628 leader_election.cc:288] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:32.444891 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72eb1c4f44934edfb49a3d6a39618df4" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:32.445017 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72eb1c4f44934edfb49a3d6a39618df4" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:32.445070 27389 raft_consensus.cc:3037] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:32.445261 27521 raft_consensus.cc:3037] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:32.447194 27389 raft_consensus.cc:2454] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:32.447273 27521 raft_consensus.cc:2454] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:32.447494 27586 leader_election.cc:302] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:32.447751 28628 raft_consensus.cc:2781] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:32.448139 28628 raft_consensus.cc:686] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:17:32.448434 28628 consensus_queue.cc:237] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:32.451037 27115 catalog_manager.cc:5484] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:32.618188 28782 raft_consensus.cc:482] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:32.618367 28782 raft_consensus.cc:504] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:32.618901 28782 leader_election.cc:288] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:32.619339 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0866b26202c14efca9a753fa1d2e8071" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:32.619370 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0866b26202c14efca9a753fa1d2e8071" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:32.619539 27651 raft_consensus.cc:2454] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:32.619575 27389 raft_consensus.cc:2454] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:32.619922 27194 leader_election.cc:302] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:32.620188 28782 raft_consensus.cc:2781] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:32.620280 28782 raft_consensus.cc:482] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:32.620400 28782 raft_consensus.cc:3037] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:32.622238 28782 raft_consensus.cc:504] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:32.622771 28782 leader_election.cc:288] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:32.623162 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0866b26202c14efca9a753fa1d2e8071" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:32.623271 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0866b26202c14efca9a753fa1d2e8071" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:32.623384 27389 raft_consensus.cc:3037] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:32.623492 27651 raft_consensus.cc:3037] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:32.626174 27389 raft_consensus.cc:2454] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:32.626188 27651 raft_consensus.cc:2454] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:32.626574 27193 leader_election.cc:302] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:32.626811 28782 raft_consensus.cc:2781] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:32.626991 28782 raft_consensus.cc:686] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:32.627243 28782 consensus_queue.cc:237] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:32.630250 27115 catalog_manager.cc:5484] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:32.911578 28628 consensus_queue.cc:1035] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:32.918428 28782 consensus_queue.cc:1035] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:32.919637 28801 consensus_queue.cc:1035] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:32.926326 28743 consensus_queue.cc:1035] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:33.012975 28804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:33.013267 28804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:33.021203 28804 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 08:17:33.069154 28782 consensus_queue.cc:1035] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:33.073251 28743 consensus_queue.cc:1035] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 218.194 ms
time per row: 0.109097 ms
I20240502 08:17:33.385913 27108 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:33.394049 27369 tablet_service.cc:1467] Processing CreateTablet for tablet a793eaaeba89458baf8af7caf3f24c70 (DEFAULT_TABLE table=rebalance_test_table_7 [id=83c398ec83c74ebf84113a0334649e63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:33.394371 27237 tablet_service.cc:1467] Processing CreateTablet for tablet a793eaaeba89458baf8af7caf3f24c70 (DEFAULT_TABLE table=rebalance_test_table_7 [id=83c398ec83c74ebf84113a0334649e63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:33.394565 27369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a793eaaeba89458baf8af7caf3f24c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:33.394855 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a793eaaeba89458baf8af7caf3f24c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:33.394945 27500 tablet_service.cc:1467] Processing CreateTablet for tablet a793eaaeba89458baf8af7caf3f24c70 (DEFAULT_TABLE table=rebalance_test_table_7 [id=83c398ec83c74ebf84113a0334649e63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:33.395376 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a793eaaeba89458baf8af7caf3f24c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:33.396260 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 6978aa3955504c548778066ddd0910e5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=83c398ec83c74ebf84113a0334649e63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:33.396644 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6978aa3955504c548778066ddd0910e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:33.398363 27498 tablet_service.cc:1467] Processing CreateTablet for tablet 6978aa3955504c548778066ddd0910e5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=83c398ec83c74ebf84113a0334649e63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:33.398715 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6978aa3955504c548778066ddd0910e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:33.404013 27368 tablet_service.cc:1467] Processing CreateTablet for tablet b82841ff10f24cf7a0cf490be3defe69 (DEFAULT_TABLE table=rebalance_test_table_7 [id=83c398ec83c74ebf84113a0334649e63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:33.404502 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82841ff10f24cf7a0cf490be3defe69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:33.404888 28869 tablet_bootstrap.cc:492] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:33.406035 27629 tablet_service.cc:1467] Processing CreateTablet for tablet 6978aa3955504c548778066ddd0910e5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=83c398ec83c74ebf84113a0334649e63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:33.406489 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6978aa3955504c548778066ddd0910e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:33.409329 27630 tablet_service.cc:1467] Processing CreateTablet for tablet b82841ff10f24cf7a0cf490be3defe69 (DEFAULT_TABLE table=rebalance_test_table_7 [id=83c398ec83c74ebf84113a0334649e63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:33.409664 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82841ff10f24cf7a0cf490be3defe69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:33.413336 27499 tablet_service.cc:1467] Processing CreateTablet for tablet b82841ff10f24cf7a0cf490be3defe69 (DEFAULT_TABLE table=rebalance_test_table_7 [id=83c398ec83c74ebf84113a0334649e63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:33.413810 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82841ff10f24cf7a0cf490be3defe69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:33.418617 28871 tablet_bootstrap.cc:492] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:33.421146 28871 tablet_bootstrap.cc:654] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:33.421849 28868 tablet_bootstrap.cc:492] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:33.421897 28870 tablet_bootstrap.cc:492] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:33.423580 28869 tablet_bootstrap.cc:654] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:33.424074 28868 tablet_bootstrap.cc:654] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:33.424575 28870 tablet_bootstrap.cc:654] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:33.430343 28871 tablet_bootstrap.cc:492] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:33.430584 28871 ts_tablet_manager.cc:1367] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20240502 08:17:33.431679 28871 raft_consensus.cc:348] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:33.432111 28871 raft_consensus.cc:374] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:33.432277 28871 raft_consensus.cc:729] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:33.432600 28871 consensus_queue.cc:260] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:33.432865 28869 tablet_bootstrap.cc:492] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:33.433014 28869 ts_tablet_manager.cc:1367] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.005s
I20240502 08:17:33.433277 28871 ts_tablet_manager.cc:1398] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:33.433687 28871 tablet_bootstrap.cc:492] T 6978aa3955504c548778066ddd0910e5 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:33.433739 28868 tablet_bootstrap.cc:492] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:33.433764 28869 raft_consensus.cc:348] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:33.433888 28868 ts_tablet_manager.cc:1367] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240502 08:17:33.433987 28869 raft_consensus.cc:374] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:33.434127 28869 raft_consensus.cc:729] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:33.434449 28869 consensus_queue.cc:260] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:33.434664 28868 raft_consensus.cc:348] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:33.434895 28868 raft_consensus.cc:374] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:33.434988 28868 raft_consensus.cc:729] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:33.435247 28868 consensus_queue.cc:260] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:33.435453 28871 tablet_bootstrap.cc:654] T 6978aa3955504c548778066ddd0910e5 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:33.435770 28868 ts_tablet_manager.cc:1398] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:33.436641 28869 ts_tablet_manager.cc:1398] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:33.436479 28870 tablet_bootstrap.cc:492] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:33.437039 28869 tablet_bootstrap.cc:492] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:33.437112 28870 ts_tablet_manager.cc:1367] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20240502 08:17:33.437894 28868 tablet_bootstrap.cc:492] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:33.439572 28870 raft_consensus.cc:348] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:33.440757 28869 tablet_bootstrap.cc:654] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:33.440934 28868 tablet_bootstrap.cc:654] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:33.441061 28870 raft_consensus.cc:374] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:33.441195 28870 raft_consensus.cc:729] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:33.441462 28870 consensus_queue.cc:260] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:33.442065 28870 ts_tablet_manager.cc:1398] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:17:33.442942 28870 tablet_bootstrap.cc:492] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:33.445003 28870 tablet_bootstrap.cc:654] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:33.458071 28870 tablet_bootstrap.cc:492] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:33.458326 28870 ts_tablet_manager.cc:1367] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.008s
I20240502 08:17:33.458750 28871 tablet_bootstrap.cc:492] T 6978aa3955504c548778066ddd0910e5 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:33.458992 28871 ts_tablet_manager.cc:1367] T 6978aa3955504c548778066ddd0910e5 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.016s
I20240502 08:17:33.459287 28869 tablet_bootstrap.cc:492] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:33.459230 28870 raft_consensus.cc:348] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:33.459486 28868 tablet_bootstrap.cc:492] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:33.459494 28869 ts_tablet_manager.cc:1367] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.008s
I20240502 08:17:33.459540 28870 raft_consensus.cc:374] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:33.459693 28868 ts_tablet_manager.cc:1367] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.002s
I20240502 08:17:33.459687 28870 raft_consensus.cc:729] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:33.459995 28870 consensus_queue.cc:260] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:33.460003 28871 raft_consensus.cc:348] T 6978aa3955504c548778066ddd0910e5 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:33.460206 28871 raft_consensus.cc:374] T 6978aa3955504c548778066ddd0910e5 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:33.460212 28869 raft_consensus.cc:348] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:33.460317 28871 raft_consensus.cc:729] T 6978aa3955504c548778066ddd0910e5 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:33.460460 28869 raft_consensus.cc:374] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:33.460568 28869 raft_consensus.cc:729] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:33.460589 28871 consensus_queue.cc:260] T 6978aa3955504c548778066ddd0910e5 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:33.460812 28870 ts_tablet_manager.cc:1398] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:33.460857 28869 consensus_queue.cc:260] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:33.461298 28871 ts_tablet_manager.cc:1398] T 6978aa3955504c548778066ddd0910e5 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:33.461582 28869 ts_tablet_manager.cc:1398] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:33.461828 28870 tablet_bootstrap.cc:492] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:33.462284 28868 raft_consensus.cc:348] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:33.462544 28868 raft_consensus.cc:374] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:33.462664 28868 raft_consensus.cc:729] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:33.463563 28868 consensus_queue.cc:260] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:33.464337 28870 tablet_bootstrap.cc:654] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:33.465114 28868 ts_tablet_manager.cc:1398] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:17:33.468674 28870 tablet_bootstrap.cc:492] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:33.468797 28870 ts_tablet_manager.cc:1367] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240502 08:17:33.469364 28870 raft_consensus.cc:348] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:33.469501 28870 raft_consensus.cc:374] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:33.469563 28870 raft_consensus.cc:729] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:33.469745 28870 consensus_queue.cc:260] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:33.470263 28870 ts_tablet_manager.cc:1398] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:17:33.541461 28743 raft_consensus.cc:482] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:33.541630 28743 raft_consensus.cc:504] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:33.542429 28743 leader_election.cc:288] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:33.542831 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978aa3955504c548778066ddd0910e5" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:33.542860 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978aa3955504c548778066ddd0910e5" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:33.543064 27520 raft_consensus.cc:2454] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:33.543152 27651 raft_consensus.cc:2454] T 6978aa3955504c548778066ddd0910e5 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:33.543481 27194 leader_election.cc:302] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:33.543715 28743 raft_consensus.cc:2781] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:33.543838 28743 raft_consensus.cc:482] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:33.543928 28743 raft_consensus.cc:3037] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:33.545926 28743 raft_consensus.cc:504] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:33.546501 28743 leader_election.cc:288] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:33.546841 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978aa3955504c548778066ddd0910e5" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:33.547037 27520 raft_consensus.cc:3037] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:33.546985 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978aa3955504c548778066ddd0910e5" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:33.547191 27651 raft_consensus.cc:3037] T 6978aa3955504c548778066ddd0910e5 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:33.548849 27520 raft_consensus.cc:2454] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:33.548923 27651 raft_consensus.cc:2454] T 6978aa3955504c548778066ddd0910e5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:33.549173 27194 leader_election.cc:302] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:33.549422 28743 raft_consensus.cc:2781] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:33.549541 28743 raft_consensus.cc:686] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:33.549779 28743 consensus_queue.cc:237] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:33.552691 27115 catalog_manager.cc:5484] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:33.573076 28743 raft_consensus.cc:482] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:33.573273 28743 raft_consensus.cc:504] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:33.573877 28743 leader_election.cc:288] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:33.574424 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a793eaaeba89458baf8af7caf3f24c70" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:33.574425 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a793eaaeba89458baf8af7caf3f24c70" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:33.574666 27520 raft_consensus.cc:2454] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:33.574666 27390 raft_consensus.cc:2454] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:33.575064 27194 leader_election.cc:302] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:33.575382 28743 raft_consensus.cc:2781] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:33.575474 28743 raft_consensus.cc:482] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:33.575573 28743 raft_consensus.cc:3037] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:33.577598 28743 raft_consensus.cc:504] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:33.578207 28743 leader_election.cc:288] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:33.578550 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a793eaaeba89458baf8af7caf3f24c70" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:33.578599 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a793eaaeba89458baf8af7caf3f24c70" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:33.578730 27390 raft_consensus.cc:3037] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:33.578760 27520 raft_consensus.cc:3037] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:33.580507 27520 raft_consensus.cc:2454] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:33.580561 27390 raft_consensus.cc:2454] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:33.580871 27194 leader_election.cc:302] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:33.581174 28743 raft_consensus.cc:2781] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:33.581336 28743 raft_consensus.cc:686] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:33.581580 28743 consensus_queue.cc:237] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:33.584200 27115 catalog_manager.cc:5484] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:33.662060 28681 raft_consensus.cc:482] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:33.662201 28681 raft_consensus.cc:504] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:33.662726 28681 leader_election.cc:288] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:33.663066 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82841ff10f24cf7a0cf490be3defe69" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:33.663113 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82841ff10f24cf7a0cf490be3defe69" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:33.663246 27390 raft_consensus.cc:2454] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:33.663331 27651 raft_consensus.cc:2454] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:33.663702 27455 leader_election.cc:302] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:33.664005 28681 raft_consensus.cc:2781] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:33.664129 28681 raft_consensus.cc:482] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:33.664214 28681 raft_consensus.cc:3037] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:33.666564 28681 raft_consensus.cc:504] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:33.666991 28681 leader_election.cc:288] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:33.667333 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82841ff10f24cf7a0cf490be3defe69" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:33.667563 27390 raft_consensus.cc:3037] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:33.667557 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82841ff10f24cf7a0cf490be3defe69" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:33.667747 27651 raft_consensus.cc:3037] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:33.670459 27390 raft_consensus.cc:2454] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:33.670534 27651 raft_consensus.cc:2454] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:33.670820 27455 leader_election.cc:302] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:33.671214 28681 raft_consensus.cc:2781] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:33.671361 28681 raft_consensus.cc:686] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:17:33.671603 28681 consensus_queue.cc:237] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:33.674276 27115 catalog_manager.cc:5484] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:33.993867 28743 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:34.001894 28743 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:34.044425 28884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:34.044725 28884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:34.052340 28884 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 08:17:34.080797 28743 consensus_queue.cc:1035] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:34.085829 28887 consensus_queue.cc:1035] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:34.091665 28681 consensus_queue.cc:1035] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:34.096002 28808 consensus_queue.cc:1035] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 232.179 ms
time per row: 0.11609 ms
I20240502 08:17:34.435770 27108 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:34.444022 27498 tablet_service.cc:1467] Processing CreateTablet for tablet f11f7ab800ba4466acb6cf9fefd19c52 (DEFAULT_TABLE table=rebalance_test_table_8 [id=9eb13e13b76544e79bcde9a39e3b2cd4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:34.444022 27237 tablet_service.cc:1467] Processing CreateTablet for tablet f11f7ab800ba4466acb6cf9fefd19c52 (DEFAULT_TABLE table=rebalance_test_table_8 [id=9eb13e13b76544e79bcde9a39e3b2cd4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:34.444466 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11f7ab800ba4466acb6cf9fefd19c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:34.444463 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11f7ab800ba4466acb6cf9fefd19c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:34.445396 27499 tablet_service.cc:1467] Processing CreateTablet for tablet 14446aeb355e4b68bcc81da74d22ad9b (DEFAULT_TABLE table=rebalance_test_table_8 [id=9eb13e13b76544e79bcde9a39e3b2cd4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:34.445555 27369 tablet_service.cc:1467] Processing CreateTablet for tablet f11f7ab800ba4466acb6cf9fefd19c52 (DEFAULT_TABLE table=rebalance_test_table_8 [id=9eb13e13b76544e79bcde9a39e3b2cd4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:34.445802 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14446aeb355e4b68bcc81da74d22ad9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:34.445956 27369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11f7ab800ba4466acb6cf9fefd19c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:34.446728 27238 tablet_service.cc:1467] Processing CreateTablet for tablet fbec19764eb84f1ca920a785d8b56d8b (DEFAULT_TABLE table=rebalance_test_table_8 [id=9eb13e13b76544e79bcde9a39e3b2cd4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:34.447072 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbec19764eb84f1ca920a785d8b56d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:34.450574 27630 tablet_service.cc:1467] Processing CreateTablet for tablet fbec19764eb84f1ca920a785d8b56d8b (DEFAULT_TABLE table=rebalance_test_table_8 [id=9eb13e13b76544e79bcde9a39e3b2cd4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:34.450948 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbec19764eb84f1ca920a785d8b56d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:34.451607 27368 tablet_service.cc:1467] Processing CreateTablet for tablet fbec19764eb84f1ca920a785d8b56d8b (DEFAULT_TABLE table=rebalance_test_table_8 [id=9eb13e13b76544e79bcde9a39e3b2cd4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:34.451992 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbec19764eb84f1ca920a785d8b56d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:34.454368 27629 tablet_service.cc:1467] Processing CreateTablet for tablet 14446aeb355e4b68bcc81da74d22ad9b (DEFAULT_TABLE table=rebalance_test_table_8 [id=9eb13e13b76544e79bcde9a39e3b2cd4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:34.454839 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14446aeb355e4b68bcc81da74d22ad9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:34.456486 28953 tablet_bootstrap.cc:492] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:34.458410 28953 tablet_bootstrap.cc:654] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:34.458518 27236 tablet_service.cc:1467] Processing CreateTablet for tablet 14446aeb355e4b68bcc81da74d22ad9b (DEFAULT_TABLE table=rebalance_test_table_8 [id=9eb13e13b76544e79bcde9a39e3b2cd4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:34.459013 27236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14446aeb355e4b68bcc81da74d22ad9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:34.464048 28953 tablet_bootstrap.cc:492] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:34.464226 28953 ts_tablet_manager.cc:1367] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 08:17:34.464258 28952 tablet_bootstrap.cc:492] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:34.465322 28953 raft_consensus.cc:348] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:34.465595 28953 raft_consensus.cc:374] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:34.465723 28953 raft_consensus.cc:729] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:34.466425 28952 tablet_bootstrap.cc:654] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:34.465988 28953 consensus_queue.cc:260] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:34.467355 28953 ts_tablet_manager.cc:1398] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:34.467726 28953 tablet_bootstrap.cc:492] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:34.471112 28953 tablet_bootstrap.cc:654] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:34.472342 28956 tablet_bootstrap.cc:492] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:34.473953 28956 tablet_bootstrap.cc:654] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:34.482177 28953 tablet_bootstrap.cc:492] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:34.482391 28953 ts_tablet_manager.cc:1367] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20240502 08:17:34.483078 28956 tablet_bootstrap.cc:492] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:34.483688 28956 ts_tablet_manager.cc:1367] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 08:17:34.484220 28953 raft_consensus.cc:348] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:34.484498 28953 raft_consensus.cc:374] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:34.484586 28953 raft_consensus.cc:729] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:34.484869 28953 consensus_queue.cc:260] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:34.485268 28956 raft_consensus.cc:348] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:34.485857 28956 raft_consensus.cc:374] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:34.486081 28956 raft_consensus.cc:729] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:34.486802 28956 consensus_queue.cc:260] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:34.489485 28953 ts_tablet_manager.cc:1398] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:17:34.490051 28951 tablet_bootstrap.cc:492] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:34.491770 28952 tablet_bootstrap.cc:492] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:34.491901 28952 ts_tablet_manager.cc:1367] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.013s
I20240502 08:17:34.492287 28956 ts_tablet_manager.cc:1398] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:17:34.492444 28951 tablet_bootstrap.cc:654] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:34.492592 28956 tablet_bootstrap.cc:492] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:34.492617 28952 raft_consensus.cc:348] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:34.492812 28952 raft_consensus.cc:374] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:34.492898 28952 raft_consensus.cc:729] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:34.493119 28952 consensus_queue.cc:260] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:34.494382 28952 ts_tablet_manager.cc:1398] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:34.494426 28956 tablet_bootstrap.cc:654] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:34.494668 28952 tablet_bootstrap.cc:492] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:34.496304 28952 tablet_bootstrap.cc:654] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:34.501637 28952 tablet_bootstrap.cc:492] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:34.501752 28952 ts_tablet_manager.cc:1367] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20240502 08:17:34.502431 28952 raft_consensus.cc:348] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:34.502573 28952 raft_consensus.cc:374] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:34.502631 28952 raft_consensus.cc:729] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:34.502665 28956 tablet_bootstrap.cc:492] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:34.502815 28956 ts_tablet_manager.cc:1367] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240502 08:17:34.502884 28952 consensus_queue.cc:260] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:34.503439 28956 raft_consensus.cc:348] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:34.503654 28956 raft_consensus.cc:374] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:34.503734 28956 raft_consensus.cc:729] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:34.503895 28952 ts_tablet_manager.cc:1398] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:34.503963 28956 consensus_queue.cc:260] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:34.504487 28956 ts_tablet_manager.cc:1398] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:34.504988 28951 tablet_bootstrap.cc:492] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:34.505133 28951 ts_tablet_manager.cc:1367] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20240502 08:17:34.505961 28951 raft_consensus.cc:348] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:34.506217 28951 raft_consensus.cc:374] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:34.506316 28951 raft_consensus.cc:729] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:34.506628 28951 consensus_queue.cc:260] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:34.507094 28951 ts_tablet_manager.cc:1398] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:34.507427 28951 tablet_bootstrap.cc:492] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:34.508898 28951 tablet_bootstrap.cc:654] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:34.510811 28951 tablet_bootstrap.cc:492] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:34.510995 28951 ts_tablet_manager.cc:1367] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240502 08:17:34.511684 28951 raft_consensus.cc:348] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:34.512096 28951 raft_consensus.cc:374] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:34.512204 28951 raft_consensus.cc:729] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:34.512418 28951 consensus_queue.cc:260] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:34.512864 28951 ts_tablet_manager.cc:1398] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:34.513188 28951 tablet_bootstrap.cc:492] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:34.514674 28951 tablet_bootstrap.cc:654] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:34.516752 28951 tablet_bootstrap.cc:492] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:34.516903 28951 ts_tablet_manager.cc:1367] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 08:17:34.517602 28951 raft_consensus.cc:348] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:34.517741 28951 raft_consensus.cc:374] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:34.517808 28951 raft_consensus.cc:729] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:34.517968 28951 consensus_queue.cc:260] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:34.518362 28951 ts_tablet_manager.cc:1398] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:17:34.565301 28858 raft_consensus.cc:482] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:34.565479 28858 raft_consensus.cc:504] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:34.566056 28858 leader_election.cc:288] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:34.566495 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec19764eb84f1ca920a785d8b56d8b" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:34.566602 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec19764eb84f1ca920a785d8b56d8b" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:34.566716 27257 raft_consensus.cc:2454] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:34.566807 27387 raft_consensus.cc:2454] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:34.567082 27586 leader_election.cc:302] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:34.567306 28858 raft_consensus.cc:2781] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:34.567414 28858 raft_consensus.cc:482] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:34.567494 28858 raft_consensus.cc:3037] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:34.569331 28858 raft_consensus.cc:504] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:34.569825 28858 leader_election.cc:288] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:34.570174 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec19764eb84f1ca920a785d8b56d8b" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:34.570372 27257 raft_consensus.cc:3037] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:34.570609 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec19764eb84f1ca920a785d8b56d8b" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:34.570770 27387 raft_consensus.cc:3037] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:34.572216 27257 raft_consensus.cc:2454] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:34.572503 27387 raft_consensus.cc:2454] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:34.572577 27586 leader_election.cc:302] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:34.572830 28858 raft_consensus.cc:2781] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:34.573215 28858 raft_consensus.cc:686] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:17:34.573494 28858 consensus_queue.cc:237] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:34.576185 27109 catalog_manager.cc:5484] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:34.639389 28858 raft_consensus.cc:482] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:34.639572 28858 raft_consensus.cc:504] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:34.640136 28858 leader_election.cc:288] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:34.640509 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14446aeb355e4b68bcc81da74d22ad9b" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:34.640664 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14446aeb355e4b68bcc81da74d22ad9b" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:34.640765 27520 raft_consensus.cc:2454] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:34.640882 27257 raft_consensus.cc:2454] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:34.641209 27587 leader_election.cc:302] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:34.641474 28858 raft_consensus.cc:2781] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:34.641569 28858 raft_consensus.cc:482] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:34.641678 28858 raft_consensus.cc:3037] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:34.643640 28858 raft_consensus.cc:504] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:34.644325 28858 leader_election.cc:288] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:34.644654 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14446aeb355e4b68bcc81da74d22ad9b" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:34.644728 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14446aeb355e4b68bcc81da74d22ad9b" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:34.644832 27521 raft_consensus.cc:3037] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:34.644932 27257 raft_consensus.cc:3037] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:34.646889 27257 raft_consensus.cc:2454] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:34.646920 27521 raft_consensus.cc:2454] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:34.647260 27586 leader_election.cc:302] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:34.647497 28858 raft_consensus.cc:2781] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:34.647634 28858 raft_consensus.cc:686] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:17:34.647858 28858 consensus_queue.cc:237] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:34.650465 27109 catalog_manager.cc:5484] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:34.728662 28887 raft_consensus.cc:482] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:34.728868 28887 raft_consensus.cc:504] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:34.729401 28887 leader_election.cc:288] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:34.729755 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11f7ab800ba4466acb6cf9fefd19c52" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:34.729992 27521 raft_consensus.cc:2454] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:34.729943 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11f7ab800ba4466acb6cf9fefd19c52" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:34.730194 27387 raft_consensus.cc:2454] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:34.730412 27194 leader_election.cc:302] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:34.730731 28887 raft_consensus.cc:2781] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:34.730827 28887 raft_consensus.cc:482] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:34.730930 28887 raft_consensus.cc:3037] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:34.732925 28887 raft_consensus.cc:504] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:34.733830 28887 leader_election.cc:288] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:34.733879 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11f7ab800ba4466acb6cf9fefd19c52" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:34.734093 27387 raft_consensus.cc:3037] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:34.736896 27387 raft_consensus.cc:2454] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:34.737259 27193 leader_election.cc:302] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:34.737581 28887 raft_consensus.cc:2781] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:34.737814 28887 raft_consensus.cc:686] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:34.738075 28887 consensus_queue.cc:237] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:34.738993 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11f7ab800ba4466acb6cf9fefd19c52" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:34.739256 27521 raft_consensus.cc:3037] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:34.741376 27521 raft_consensus.cc:2454] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:34.746954 27109 catalog_manager.cc:5484] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:34.964100 28858 consensus_queue.cc:1035] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:34.972071 28858 consensus_queue.cc:1035] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:35.093000 28968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:35.093304 28968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:35.100481 28968 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:17:35.109556 28858 consensus_queue.cc:1035] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:35.115305 28972 consensus_queue.cc:1035] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240502 08:17:35.236236 28743 consensus_queue.cc:1035] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:35.260185 28743 consensus_queue.cc:1035] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 237.687 ms
time per row: 0.118844 ms
I20240502 08:17:35.501652 27110 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:35.509902 27498 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0313e85d444e098bfa35be89a3ad97 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a65036830fa64a22b9f4efd6d5bd4f9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:35.510135 27630 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0313e85d444e098bfa35be89a3ad97 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a65036830fa64a22b9f4efd6d5bd4f9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:35.510396 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0313e85d444e098bfa35be89a3ad97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.510545 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0313e85d444e098bfa35be89a3ad97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.510888 27369 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0313e85d444e098bfa35be89a3ad97 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a65036830fa64a22b9f4efd6d5bd4f9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:35.511283 27369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0313e85d444e098bfa35be89a3ad97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.512755 27500 tablet_service.cc:1467] Processing CreateTablet for tablet 09aa1b5d99114b3095be08c9547fbb54 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a65036830fa64a22b9f4efd6d5bd4f9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:35.513053 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09aa1b5d99114b3095be08c9547fbb54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.514390 27499 tablet_service.cc:1467] Processing CreateTablet for tablet 33aee3df5eb242ef9cc056d25f6d6a82 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a65036830fa64a22b9f4efd6d5bd4f9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:35.514750 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33aee3df5eb242ef9cc056d25f6d6a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.518474 27368 tablet_service.cc:1467] Processing CreateTablet for tablet 33aee3df5eb242ef9cc056d25f6d6a82 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a65036830fa64a22b9f4efd6d5bd4f9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:35.518977 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33aee3df5eb242ef9cc056d25f6d6a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.521085 27370 tablet_service.cc:1467] Processing CreateTablet for tablet 09aa1b5d99114b3095be08c9547fbb54 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a65036830fa64a22b9f4efd6d5bd4f9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:35.521360 29037 tablet_bootstrap.cc:492] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:35.521531 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09aa1b5d99114b3095be08c9547fbb54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.522135 27629 tablet_service.cc:1467] Processing CreateTablet for tablet 33aee3df5eb242ef9cc056d25f6d6a82 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a65036830fa64a22b9f4efd6d5bd4f9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:35.522501 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33aee3df5eb242ef9cc056d25f6d6a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.524513 27237 tablet_service.cc:1467] Processing CreateTablet for tablet 09aa1b5d99114b3095be08c9547fbb54 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a65036830fa64a22b9f4efd6d5bd4f9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:35.524950 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09aa1b5d99114b3095be08c9547fbb54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.530704 29036 tablet_bootstrap.cc:492] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:35.531788 29037 tablet_bootstrap.cc:654] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:35.532397 29036 tablet_bootstrap.cc:654] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:35.536664 29041 tablet_bootstrap.cc:492] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:35.537318 29038 tablet_bootstrap.cc:492] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:35.538990 29041 tablet_bootstrap.cc:654] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:35.539151 29038 tablet_bootstrap.cc:654] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:35.541525 29037 tablet_bootstrap.cc:492] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:35.541988 29036 tablet_bootstrap.cc:492] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:35.542248 29036 ts_tablet_manager.cc:1367] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240502 08:17:35.542327 29037 ts_tablet_manager.cc:1367] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.005s
I20240502 08:17:35.543298 29036 raft_consensus.cc:348] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:35.543571 29038 tablet_bootstrap.cc:492] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:35.543474 29037 raft_consensus.cc:348] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:35.543658 29037 raft_consensus.cc:374] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:35.543689 29038 ts_tablet_manager.cc:1367] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240502 08:17:35.543772 29036 raft_consensus.cc:374] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:35.543835 29037 raft_consensus.cc:729] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:35.543958 29036 raft_consensus.cc:729] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:35.544137 29037 consensus_queue.cc:260] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:35.544420 29038 raft_consensus.cc:348] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:35.544306 29036 consensus_queue.cc:260] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:35.544625 29038 raft_consensus.cc:374] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:35.544709 29038 raft_consensus.cc:729] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:35.544896 29038 consensus_queue.cc:260] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:35.546027 29037 ts_tablet_manager.cc:1398] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:17:35.546533 29037 tablet_bootstrap.cc:492] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:35.547091 29041 tablet_bootstrap.cc:492] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:35.547240 29041 ts_tablet_manager.cc:1367] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240502 08:17:35.547298 29036 ts_tablet_manager.cc:1398] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:17:35.547578 29036 tablet_bootstrap.cc:492] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:35.545531 29038 ts_tablet_manager.cc:1398] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 08:17:35.547962 29041 raft_consensus.cc:348] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:35.548157 29041 raft_consensus.cc:374] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:35.548247 29041 raft_consensus.cc:729] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:35.548280 29038 tablet_bootstrap.cc:492] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:35.548511 29041 consensus_queue.cc:260] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:35.549954 29041 ts_tablet_manager.cc:1398] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:17:35.550040 29036 tablet_bootstrap.cc:654] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:35.551633 29036 tablet_bootstrap.cc:492] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:35.552361 29038 tablet_bootstrap.cc:654] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:35.552590 29036 ts_tablet_manager.cc:1367] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240502 08:17:35.553333 29036 raft_consensus.cc:348] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:35.553526 29036 raft_consensus.cc:374] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:35.553591 29036 raft_consensus.cc:729] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:35.553807 29036 consensus_queue.cc:260] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:35.554296 29036 ts_tablet_manager.cc:1398] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:35.554585 29036 tablet_bootstrap.cc:492] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:35.555831 29037 tablet_bootstrap.cc:654] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:35.556327 29036 tablet_bootstrap.cc:654] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:35.567577 29038 tablet_bootstrap.cc:492] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:35.567752 29038 ts_tablet_manager.cc:1367] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20240502 08:17:35.567764 29036 tablet_bootstrap.cc:492] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:35.567893 29036 ts_tablet_manager.cc:1367] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20240502 08:17:35.567974 28929 raft_consensus.cc:482] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:35.568130 28929 raft_consensus.cc:504] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:35.568439 29037 tablet_bootstrap.cc:492] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:35.568521 29038 raft_consensus.cc:348] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:35.568656 29037 ts_tablet_manager.cc:1367] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.010s
I20240502 08:17:35.568734 29038 raft_consensus.cc:374] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:35.568637 29036 raft_consensus.cc:348] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:35.568804 29038 raft_consensus.cc:729] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:35.568825 28929 leader_election.cc:288] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:35.568842 29036 raft_consensus.cc:374] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:35.569116 29036 raft_consensus.cc:729] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:35.569038 29038 consensus_queue.cc:260] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:35.569344 29036 consensus_queue.cc:260] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:35.569551 29038 ts_tablet_manager.cc:1398] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:35.570154 29036 ts_tablet_manager.cc:1398] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 08:17:35.570590 29037 raft_consensus.cc:348] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:35.570778 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09aa1b5d99114b3095be08c9547fbb54" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:35.570812 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09aa1b5d99114b3095be08c9547fbb54" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:35.570895 29037 raft_consensus.cc:374] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:35.571008 27257 raft_consensus.cc:2454] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:35.571079 29037 raft_consensus.cc:729] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
W20240502 08:17:35.571337 27455 leader_election.cc:341] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:35.571391 29037 consensus_queue.cc:260] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:35.571678 27455 leader_election.cc:302] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters: 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:17:35.572031 28929 raft_consensus.cc:2781] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:35.572052 29037 ts_tablet_manager.cc:1398] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:17:35.572150 28929 raft_consensus.cc:482] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:35.572225 28929 raft_consensus.cc:3037] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:35.572348 29037 tablet_bootstrap.cc:492] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:35.574961 28929 raft_consensus.cc:504] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:35.575222 29037 tablet_bootstrap.cc:654] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:35.575609 28929 leader_election.cc:288] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:35.577230 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09aa1b5d99114b3095be08c9547fbb54" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:35.577718 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09aa1b5d99114b3095be08c9547fbb54" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:35.577927 27257 raft_consensus.cc:3037] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:35.579826 27257 raft_consensus.cc:2454] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:35.580184 27455 leader_election.cc:302] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:35.580447 29048 raft_consensus.cc:2781] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
W20240502 08:17:35.580590 27455 leader_election.cc:341] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:35.580821 29037 tablet_bootstrap.cc:492] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:35.580976 29037 ts_tablet_manager.cc:1367] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240502 08:17:35.581821 29048 raft_consensus.cc:686] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:17:35.582058 29037 raft_consensus.cc:348] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:35.582262 29037 raft_consensus.cc:374] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:35.582351 29037 raft_consensus.cc:729] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:35.582429 29048 consensus_queue.cc:237] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:35.582573 29037 consensus_queue.cc:260] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:35.583171 29037 ts_tablet_manager.cc:1398] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:35.586787 27110 catalog_manager.cc:5484] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:35.784504 29048 raft_consensus.cc:482] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:35.784698 29048 raft_consensus.cc:504] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:35.785307 29048 leader_election.cc:288] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:35.785681 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0313e85d444e098bfa35be89a3ad97" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:35.785795 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0313e85d444e098bfa35be89a3ad97" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:35.785900 27651 raft_consensus.cc:2454] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:35.786031 27390 raft_consensus.cc:2454] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:35.786306 27456 leader_election.cc:302] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:35.786593 29048 raft_consensus.cc:2781] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:35.786684 29048 raft_consensus.cc:482] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:35.786773 29048 raft_consensus.cc:3037] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:35.788513 29048 raft_consensus.cc:504] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:35.789021 29048 leader_election.cc:288] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:35.789394 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0313e85d444e098bfa35be89a3ad97" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:35.789593 27651 raft_consensus.cc:3037] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:35.789527 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0313e85d444e098bfa35be89a3ad97" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:35.789737 27390 raft_consensus.cc:3037] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:35.792273 27651 raft_consensus.cc:2454] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:35.792405 27390 raft_consensus.cc:2454] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:35.792659 27456 leader_election.cc:302] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:35.792937 29048 raft_consensus.cc:2781] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:35.793092 29048 raft_consensus.cc:686] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:17:35.793334 29048 consensus_queue.cc:237] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:35.796444 27109 catalog_manager.cc:5484] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:35.798143 28816 raft_consensus.cc:482] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:35.798283 28816 raft_consensus.cc:504] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:35.798955 28816 leader_election.cc:288] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:35.799400 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09aa1b5d99114b3095be08c9547fbb54" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:35.799649 27257 raft_consensus.cc:2379] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83e171a5e16c4008923f1d43fc5ce7b1 in current term 1: Already voted for candidate b01e7c8a302d41078fa39cfef665ff59 in this term.
I20240502 08:17:35.799952 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09aa1b5d99114b3095be08c9547fbb54" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:35.800442 27326 leader_election.cc:302] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1; no voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:17:35.800689 28816 raft_consensus.cc:3037] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:35.802558 28816 raft_consensus.cc:2726] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:35.876152 28858 raft_consensus.cc:482] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:35.876360 28858 raft_consensus.cc:504] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:35.876956 28858 leader_election.cc:288] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:35.877466 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33aee3df5eb242ef9cc056d25f6d6a82" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:35.877507 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33aee3df5eb242ef9cc056d25f6d6a82" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:35.877677 27520 raft_consensus.cc:2454] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:35.877686 27390 raft_consensus.cc:2454] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:35.878057 27586 leader_election.cc:302] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:35.878335 28858 raft_consensus.cc:2781] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:35.878417 28858 raft_consensus.cc:482] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:35.878487 28858 raft_consensus.cc:3037] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:35.880285 28858 raft_consensus.cc:504] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:35.880748 28858 leader_election.cc:288] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:35.881155 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33aee3df5eb242ef9cc056d25f6d6a82" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:35.881225 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33aee3df5eb242ef9cc056d25f6d6a82" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:35.881402 27390 raft_consensus.cc:3037] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:35.881417 27520 raft_consensus.cc:3037] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:35.884218 27390 raft_consensus.cc:2454] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:35.884238 27520 raft_consensus.cc:2454] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:35.884666 27587 leader_election.cc:302] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:35.884956 28858 raft_consensus.cc:2781] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:35.885128 28858 raft_consensus.cc:686] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:17:35.885391 28858 consensus_queue.cc:237] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:35.888029 27109 catalog_manager.cc:5484] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:35.977814 29048 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:35.983970 28929 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:36.224768 29054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:36.225045 29054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:36.232781 29054 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:17:36.251410 29048 consensus_queue.cc:1035] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:36.256536 29048 consensus_queue.cc:1035] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240502 08:17:36.291685 28858 consensus_queue.cc:1035] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:36.302525 29087 consensus_queue.cc:1035] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 266.959 ms
time per row: 0.13348 ms
I20240502 08:17:36.660521 27112 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:36.669411 27237 tablet_service.cc:1467] Processing CreateTablet for tablet fddf0ae2372b492ba67f8a4d7de47129 (DEFAULT_TABLE table=rebalance_test_table_10 [id=c9beaea1ee3d4d3b93cac242b3ff8722]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:36.669553 27629 tablet_service.cc:1467] Processing CreateTablet for tablet fddf0ae2372b492ba67f8a4d7de47129 (DEFAULT_TABLE table=rebalance_test_table_10 [id=c9beaea1ee3d4d3b93cac242b3ff8722]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:36.669863 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fddf0ae2372b492ba67f8a4d7de47129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:36.669884 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fddf0ae2372b492ba67f8a4d7de47129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:36.670991 27368 tablet_service.cc:1467] Processing CreateTablet for tablet 7e8228b23331486592b7e958a6c52f4c (DEFAULT_TABLE table=rebalance_test_table_10 [id=c9beaea1ee3d4d3b93cac242b3ff8722]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:36.670949 27370 tablet_service.cc:1467] Processing CreateTablet for tablet fddf0ae2372b492ba67f8a4d7de47129 (DEFAULT_TABLE table=rebalance_test_table_10 [id=c9beaea1ee3d4d3b93cac242b3ff8722]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:36.671478 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e8228b23331486592b7e958a6c52f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:36.671694 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fddf0ae2372b492ba67f8a4d7de47129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:36.674959 27498 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf33bf3e4d542f287ba670c2348b34d (DEFAULT_TABLE table=rebalance_test_table_10 [id=c9beaea1ee3d4d3b93cac242b3ff8722]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:36.675215 27630 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf33bf3e4d542f287ba670c2348b34d (DEFAULT_TABLE table=rebalance_test_table_10 [id=c9beaea1ee3d4d3b93cac242b3ff8722]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:36.675390 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf33bf3e4d542f287ba670c2348b34d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:36.675576 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf33bf3e4d542f287ba670c2348b34d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:36.678683 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 7e8228b23331486592b7e958a6c52f4c (DEFAULT_TABLE table=rebalance_test_table_10 [id=c9beaea1ee3d4d3b93cac242b3ff8722]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:36.678771 27236 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf33bf3e4d542f287ba670c2348b34d (DEFAULT_TABLE table=rebalance_test_table_10 [id=c9beaea1ee3d4d3b93cac242b3ff8722]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:36.679150 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e8228b23331486592b7e958a6c52f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:36.679661 27236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf33bf3e4d542f287ba670c2348b34d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:36.682368 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 7e8228b23331486592b7e958a6c52f4c (DEFAULT_TABLE table=rebalance_test_table_10 [id=c9beaea1ee3d4d3b93cac242b3ff8722]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:36.682718 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e8228b23331486592b7e958a6c52f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:36.696003 29120 tablet_bootstrap.cc:492] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:36.698283 29120 tablet_bootstrap.cc:654] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:36.702291 29122 tablet_bootstrap.cc:492] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:36.703037 29120 tablet_bootstrap.cc:492] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:36.703248 29120 ts_tablet_manager.cc:1367] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 08:17:36.704120 29122 tablet_bootstrap.cc:654] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:36.704033 29120 raft_consensus.cc:348] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:36.704295 29120 raft_consensus.cc:374] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:36.704418 29120 raft_consensus.cc:729] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:36.704916 29120 consensus_queue.cc:260] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:36.705096 29123 tablet_bootstrap.cc:492] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:36.707008 29120 ts_tablet_manager.cc:1398] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:36.707311 29120 tablet_bootstrap.cc:492] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:36.707515 29121 tablet_bootstrap.cc:492] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:36.708964 29120 tablet_bootstrap.cc:654] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:36.709138 29121 tablet_bootstrap.cc:654] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:36.711006 29123 tablet_bootstrap.cc:654] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:36.711107 29120 tablet_bootstrap.cc:492] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:36.711344 29120 ts_tablet_manager.cc:1367] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:36.711740 29122 tablet_bootstrap.cc:492] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:36.711892 29122 ts_tablet_manager.cc:1367] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 08:17:36.712144 29120 raft_consensus.cc:348] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:36.712639 29122 raft_consensus.cc:348] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:36.712802 29120 raft_consensus.cc:374] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:36.712837 29122 raft_consensus.cc:374] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:36.712888 29120 raft_consensus.cc:729] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:36.712939 29122 raft_consensus.cc:729] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:36.713120 29123 tablet_bootstrap.cc:492] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:36.713131 29120 consensus_queue.cc:260] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:36.713245 29123 ts_tablet_manager.cc:1367] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240502 08:17:36.713191 29122 consensus_queue.cc:260] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:36.713865 29122 ts_tablet_manager.cc:1398] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:36.714167 29122 tablet_bootstrap.cc:492] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:36.714962 29120 ts_tablet_manager.cc:1398] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:36.715675 29122 tablet_bootstrap.cc:654] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:36.716426 29000 raft_consensus.cc:482] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:36.716322 29123 raft_consensus.cc:348] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:36.716609 29123 raft_consensus.cc:374] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:36.716560 29000 raft_consensus.cc:504] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:36.716799 29123 raft_consensus.cc:729] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:36.717077 29123 consensus_queue.cc:260] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:36.717265 29000 leader_election.cc:288] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:36.718343 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf33bf3e4d542f287ba670c2348b34d" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:36.718573 27520 raft_consensus.cc:2454] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:36.718993 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf33bf3e4d542f287ba670c2348b34d" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:36.719153 27194 leader_election.cc:302] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
W20240502 08:17:36.719599 27194 leader_election.cc:341] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:36.720449 29000 raft_consensus.cc:2781] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:36.720569 29000 raft_consensus.cc:482] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:36.720665 29000 raft_consensus.cc:3037] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:36.720693 29122 tablet_bootstrap.cc:492] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:36.720850 29122 ts_tablet_manager.cc:1367] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240502 08:17:36.721329 29123 ts_tablet_manager.cc:1398] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 08:17:36.721652 29122 raft_consensus.cc:348] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:36.721879 29122 raft_consensus.cc:374] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:36.721956 29122 raft_consensus.cc:729] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:36.722203 29122 consensus_queue.cc:260] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:36.722774 29122 ts_tablet_manager.cc:1398] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:36.722990 29122 tablet_bootstrap.cc:492] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:36.723388 29000 raft_consensus.cc:504] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:36.724236 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf33bf3e4d542f287ba670c2348b34d" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:36.724463 27520 raft_consensus.cc:3037] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:36.724959 29122 tablet_bootstrap.cc:654] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:36.726977 27520 raft_consensus.cc:2454] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:36.727138 29122 tablet_bootstrap.cc:492] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:36.728257 27194 leader_election.cc:302] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:36.728534 28743 raft_consensus.cc:2781] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:36.728958 28743 raft_consensus.cc:686] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:36.729250 28743 consensus_queue.cc:237] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:36.730062 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf33bf3e4d542f287ba670c2348b34d" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
W20240502 08:17:36.730697 27194 leader_election.cc:341] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:36.731087 29000 leader_election.cc:288] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:36.733170 29121 tablet_bootstrap.cc:492] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:36.733354 29121 ts_tablet_manager.cc:1367] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240502 08:17:36.734105 29121 raft_consensus.cc:348] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:36.734349 29121 raft_consensus.cc:374] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:36.734474 29121 raft_consensus.cc:729] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:36.734728 29121 consensus_queue.cc:260] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:36.735288 29121 ts_tablet_manager.cc:1398] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:36.735596 29121 tablet_bootstrap.cc:492] T fddf0ae2372b492ba67f8a4d7de47129 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:36.737165 29121 tablet_bootstrap.cc:654] T fddf0ae2372b492ba67f8a4d7de47129 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:36.738024 29122 ts_tablet_manager.cc:1367] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:36.738821 29122 raft_consensus.cc:348] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:36.739056 29122 raft_consensus.cc:374] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:36.739180 29122 raft_consensus.cc:729] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:36.742314 29122 consensus_queue.cc:260] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:36.743058 29122 ts_tablet_manager.cc:1398] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 08:17:36.747108 29060 raft_consensus.cc:482] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:36.747260 29060 raft_consensus.cc:504] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:36.747896 29060 leader_election.cc:288] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:36.747722 27116 catalog_manager.cc:5484] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:36.748497 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddf0ae2372b492ba67f8a4d7de47129" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
W20240502 08:17:36.749083 27326 leader_election.cc:341] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:36.749366 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddf0ae2372b492ba67f8a4d7de47129" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:36.749605 27258 raft_consensus.cc:2454] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:36.750068 27325 leader_election.cc:302] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters: bf8ac71b15f740a0a928911360773442
I20240502 08:17:36.750488 29060 raft_consensus.cc:2781] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:36.750597 29060 raft_consensus.cc:482] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:36.750679 29060 raft_consensus.cc:3037] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:36.753310 29060 raft_consensus.cc:504] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:36.754756 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddf0ae2372b492ba67f8a4d7de47129" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
W20240502 08:17:36.755297 27326 leader_election.cc:341] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:36.755604 29060 leader_election.cc:288] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:36.755756 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddf0ae2372b492ba67f8a4d7de47129" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:36.755965 27258 raft_consensus.cc:3037] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:36.756485 29121 tablet_bootstrap.cc:492] T fddf0ae2372b492ba67f8a4d7de47129 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:36.756693 29121 ts_tablet_manager.cc:1367] T fddf0ae2372b492ba67f8a4d7de47129 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.007s
I20240502 08:17:36.757448 29121 raft_consensus.cc:348] T fddf0ae2372b492ba67f8a4d7de47129 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:36.757696 29121 raft_consensus.cc:374] T fddf0ae2372b492ba67f8a4d7de47129 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:36.757692 27258 raft_consensus.cc:2454] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:36.757805 29121 raft_consensus.cc:729] T fddf0ae2372b492ba67f8a4d7de47129 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:36.758085 27325 leader_election.cc:302] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters: bf8ac71b15f740a0a928911360773442
I20240502 08:17:36.758085 29121 consensus_queue.cc:260] T fddf0ae2372b492ba67f8a4d7de47129 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:36.758507 29060 raft_consensus.cc:2781] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:36.758744 29121 ts_tablet_manager.cc:1398] T fddf0ae2372b492ba67f8a4d7de47129 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:36.759171 29121 tablet_bootstrap.cc:492] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:36.761402 29121 tablet_bootstrap.cc:654] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:36.773509 29060 raft_consensus.cc:686] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:17:36.775279 29121 tablet_bootstrap.cc:492] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:36.775458 29121 ts_tablet_manager.cc:1367] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.008s
I20240502 08:17:36.776170 29121 raft_consensus.cc:348] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:36.776407 29121 raft_consensus.cc:374] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:36.776525 29121 raft_consensus.cc:729] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:36.776803 29121 consensus_queue.cc:260] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:36.777381 29121 ts_tablet_manager.cc:1398] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:36.773806 29060 consensus_queue.cc:237] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:36.789710 29012 raft_consensus.cc:482] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:36.789844 29012 raft_consensus.cc:504] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:36.790709 29012 leader_election.cc:288] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:36.791795 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf33bf3e4d542f287ba670c2348b34d" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:36.792071 27520 raft_consensus.cc:2379] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bf8ac71b15f740a0a928911360773442 in current term 1: Already voted for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in this term.
I20240502 08:17:36.793200 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf33bf3e4d542f287ba670c2348b34d" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:36.793884 27586 leader_election.cc:302] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:17:36.794253 29012 raft_consensus.cc:3037] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:36.796617 29012 raft_consensus.cc:2726] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:36.796705 27116 catalog_manager.cc:5484] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:36.865084 29060 raft_consensus.cc:482] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:36.865285 29060 raft_consensus.cc:504] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:36.865880 29060 leader_election.cc:288] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:36.866271 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e8228b23331486592b7e958a6c52f4c" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:36.866346 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e8228b23331486592b7e958a6c52f4c" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:36.866470 27651 raft_consensus.cc:2454] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:36.866596 27258 raft_consensus.cc:2454] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:36.866919 27326 leader_election.cc:302] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:36.867203 29060 raft_consensus.cc:2781] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:36.867281 29060 raft_consensus.cc:482] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:36.867386 29060 raft_consensus.cc:3037] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:36.870275 29060 raft_consensus.cc:504] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:36.870811 29060 leader_election.cc:288] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:36.871062 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e8228b23331486592b7e958a6c52f4c" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:36.871255 27651 raft_consensus.cc:3037] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:36.871240 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e8228b23331486592b7e958a6c52f4c" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:36.871445 27258 raft_consensus.cc:3037] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:36.873839 27651 raft_consensus.cc:2454] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:36.873901 27258 raft_consensus.cc:2454] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:36.874208 27326 leader_election.cc:302] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:36.874473 29060 raft_consensus.cc:2781] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:36.874646 29060 raft_consensus.cc:686] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:17:36.874902 29060 consensus_queue.cc:237] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:36.878142 27116 catalog_manager.cc:5484] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:37.168416 29131 consensus_queue.cc:1035] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:37.171594 29060 consensus_queue.cc:1035] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:37.176076 29000 consensus_queue.cc:1035] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:37.196991 27651 raft_consensus.cc:3037] T fddf0ae2372b492ba67f8a4d7de47129 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.200526 29060 consensus_queue.cc:1035] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:37.231675 29139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:37.232053 29139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:37.243965 29139 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240502 08:17:37.390874 27257 raft_consensus.cc:1261] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:37.391465 27650 raft_consensus.cc:1261] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:37.392590 29136 consensus_queue.cc:1035] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:37.395529 29136 consensus_queue.cc:1035] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:37.436990 29147 mvcc.cc:205] Tried to move back new op lower bound from 7023156663865958400 to 7023156661761552384. Current Snapshot: MvccSnapshot[applied={T|T < 7023156663865958400}]
INSERT report
rows total: 2000
time total: 267.485 ms
time per row: 0.133742 ms
I20240502 08:17:37.653405 27112 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:37.661936 27498 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf739b314d84db08bc4cb32b0e5b0f4 (DEFAULT_TABLE table=rebalance_test_table_11 [id=f7b46161b1f6484f9449a5d5036e221f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:37.662384 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf739b314d84db08bc4cb32b0e5b0f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:37.662387 27237 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf739b314d84db08bc4cb32b0e5b0f4 (DEFAULT_TABLE table=rebalance_test_table_11 [id=f7b46161b1f6484f9449a5d5036e221f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:37.662762 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf739b314d84db08bc4cb32b0e5b0f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:37.662842 27499 tablet_service.cc:1467] Processing CreateTablet for tablet cad036d297bd45d7b7c5a1f19356906f (DEFAULT_TABLE table=rebalance_test_table_11 [id=f7b46161b1f6484f9449a5d5036e221f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:37.663177 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cad036d297bd45d7b7c5a1f19356906f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:37.666666 27368 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf739b314d84db08bc4cb32b0e5b0f4 (DEFAULT_TABLE table=rebalance_test_table_11 [id=f7b46161b1f6484f9449a5d5036e221f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:37.667078 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf739b314d84db08bc4cb32b0e5b0f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:37.668273 27630 tablet_service.cc:1467] Processing CreateTablet for tablet cad036d297bd45d7b7c5a1f19356906f (DEFAULT_TABLE table=rebalance_test_table_11 [id=f7b46161b1f6484f9449a5d5036e221f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:37.668763 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cad036d297bd45d7b7c5a1f19356906f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:37.670087 27370 tablet_service.cc:1467] Processing CreateTablet for tablet cad036d297bd45d7b7c5a1f19356906f (DEFAULT_TABLE table=rebalance_test_table_11 [id=f7b46161b1f6484f9449a5d5036e221f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:37.670563 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cad036d297bd45d7b7c5a1f19356906f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:37.674283 27236 tablet_service.cc:1467] Processing CreateTablet for tablet fef4f56761bb458c8d84807feb38a617 (DEFAULT_TABLE table=rebalance_test_table_11 [id=f7b46161b1f6484f9449a5d5036e221f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:37.674644 27236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef4f56761bb458c8d84807feb38a617. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:37.677491 27631 tablet_service.cc:1467] Processing CreateTablet for tablet fef4f56761bb458c8d84807feb38a617 (DEFAULT_TABLE table=rebalance_test_table_11 [id=f7b46161b1f6484f9449a5d5036e221f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:37.677577 27500 tablet_service.cc:1467] Processing CreateTablet for tablet fef4f56761bb458c8d84807feb38a617 (DEFAULT_TABLE table=rebalance_test_table_11 [id=f7b46161b1f6484f9449a5d5036e221f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:37.677868 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef4f56761bb458c8d84807feb38a617. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:37.677945 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef4f56761bb458c8d84807feb38a617. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:37.692982 29207 tablet_bootstrap.cc:492] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:37.695505 29207 tablet_bootstrap.cc:654] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:37.695418 29208 tablet_bootstrap.cc:492] T cad036d297bd45d7b7c5a1f19356906f P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:37.695917 29206 tablet_bootstrap.cc:492] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:37.696825 29205 tablet_bootstrap.cc:492] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:37.697393 29208 tablet_bootstrap.cc:654] T cad036d297bd45d7b7c5a1f19356906f P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:37.698235 29206 tablet_bootstrap.cc:654] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:37.698810 29205 tablet_bootstrap.cc:654] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:37.699841 29207 tablet_bootstrap.cc:492] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:37.699980 29208 tablet_bootstrap.cc:492] T cad036d297bd45d7b7c5a1f19356906f P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:37.700049 29207 ts_tablet_manager.cc:1367] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 08:17:37.700111 29208 ts_tablet_manager.cc:1367] T cad036d297bd45d7b7c5a1f19356906f P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:17:37.700663 29207 raft_consensus.cc:348] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:37.700821 29207 raft_consensus.cc:374] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:37.700842 29205 tablet_bootstrap.cc:492] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:37.700928 29207 raft_consensus.cc:729] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:37.700889 29208 raft_consensus.cc:348] T cad036d297bd45d7b7c5a1f19356906f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:37.701050 29205 ts_tablet_manager.cc:1367] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 08:17:37.701117 29208 raft_consensus.cc:374] T cad036d297bd45d7b7c5a1f19356906f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:37.701201 29208 raft_consensus.cc:729] T cad036d297bd45d7b7c5a1f19356906f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:37.701176 29207 consensus_queue.cc:260] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:37.701443 29208 consensus_queue.cc:260] T cad036d297bd45d7b7c5a1f19356906f P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:37.701813 29207 ts_tablet_manager.cc:1398] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:37.701807 29205 raft_consensus.cc:348] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:37.701995 29205 raft_consensus.cc:374] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:37.702039 29208 ts_tablet_manager.cc:1398] T cad036d297bd45d7b7c5a1f19356906f P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:37.702090 29207 tablet_bootstrap.cc:492] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:37.702317 29208 tablet_bootstrap.cc:492] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:37.703240 29207 tablet_bootstrap.cc:654] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:37.703804 29208 tablet_bootstrap.cc:654] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:37.702122 29205 raft_consensus.cc:729] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:37.704704 29205 consensus_queue.cc:260] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:37.705495 29205 ts_tablet_manager.cc:1398] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:17:37.705837 29205 tablet_bootstrap.cc:492] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:37.706632 29206 tablet_bootstrap.cc:492] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:37.706794 29206 ts_tablet_manager.cc:1367] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 08:17:37.707643 29206 raft_consensus.cc:348] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:37.707857 29206 raft_consensus.cc:374] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:37.707949 29206 raft_consensus.cc:729] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:37.708158 29206 consensus_queue.cc:260] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:37.708735 29206 ts_tablet_manager.cc:1398] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:37.709041 29206 tablet_bootstrap.cc:492] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:37.710748 29206 tablet_bootstrap.cc:654] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:37.713498 29205 tablet_bootstrap.cc:654] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:37.715826 29208 tablet_bootstrap.cc:492] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:37.716112 29208 ts_tablet_manager.cc:1367] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.005s
I20240502 08:17:37.717206 29208 raft_consensus.cc:348] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:37.717567 29208 raft_consensus.cc:374] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:37.717717 29208 raft_consensus.cc:729] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:37.718106 29208 consensus_queue.cc:260] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:37.719082 29208 ts_tablet_manager.cc:1398] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:37.720278 29206 tablet_bootstrap.cc:492] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:37.720474 29206 ts_tablet_manager.cc:1367] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240502 08:17:37.720659 29205 tablet_bootstrap.cc:492] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:37.720849 29205 ts_tablet_manager.cc:1367] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240502 08:17:37.721223 29206 raft_consensus.cc:348] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:37.721453 29206 raft_consensus.cc:374] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:37.721534 29206 raft_consensus.cc:729] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:37.721760 29206 consensus_queue.cc:260] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:37.721786 29205 raft_consensus.cc:348] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:37.722184 29205 raft_consensus.cc:374] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:37.722306 29205 raft_consensus.cc:729] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:37.722615 29205 consensus_queue.cc:260] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:37.723297 29205 ts_tablet_manager.cc:1398] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:37.724390 29206 ts_tablet_manager.cc:1398] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:37.724643 29206 tablet_bootstrap.cc:492] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:37.725930 29206 tablet_bootstrap.cc:654] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:37.729143 29206 tablet_bootstrap.cc:492] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:37.729274 29206 ts_tablet_manager.cc:1367] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:17:37.729837 29206 raft_consensus.cc:348] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:37.729991 29206 raft_consensus.cc:374] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:37.730115 29206 raft_consensus.cc:729] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:37.730309 29206 consensus_queue.cc:260] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:37.731752 29206 ts_tablet_manager.cc:1398] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:37.735060 29207 tablet_bootstrap.cc:492] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:37.735572 29207 ts_tablet_manager.cc:1367] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.017s
I20240502 08:17:37.736213 29207 raft_consensus.cc:348] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:37.736394 29207 raft_consensus.cc:374] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:37.736475 29207 raft_consensus.cc:729] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:37.736708 29207 consensus_queue.cc:260] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:37.737300 29207 ts_tablet_manager.cc:1398] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:37.742862 29190 raft_consensus.cc:482] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:37.743011 29190 raft_consensus.cc:504] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:37.743687 29190 leader_election.cc:288] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:37.744534 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf739b314d84db08bc4cb32b0e5b0f4" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:37.744562 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf739b314d84db08bc4cb32b0e5b0f4" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:37.744711 27258 raft_consensus.cc:2454] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:37.744760 27520 raft_consensus.cc:2454] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:37.745147 27326 leader_election.cc:302] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:37.745405 29190 raft_consensus.cc:2781] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:37.745512 29190 raft_consensus.cc:482] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:37.745616 29190 raft_consensus.cc:3037] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.747922 29190 raft_consensus.cc:504] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:37.748754 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf739b314d84db08bc4cb32b0e5b0f4" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:37.748926 27520 raft_consensus.cc:3037] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.748415 29190 leader_election.cc:288] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:37.749583 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf739b314d84db08bc4cb32b0e5b0f4" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:37.749755 27258 raft_consensus.cc:3037] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.751636 27520 raft_consensus.cc:2454] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:37.752000 27326 leader_election.cc:302] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:37.752274 29190 raft_consensus.cc:2781] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:37.752413 29190 raft_consensus.cc:686] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:17:37.752676 27258 raft_consensus.cc:2454] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:37.752715 29190 consensus_queue.cc:237] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:37.756204 27112 catalog_manager.cc:5484] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:37.898272 28929 raft_consensus.cc:482] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:37.898469 28929 raft_consensus.cc:504] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:37.899009 28929 leader_election.cc:288] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:37.899447 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef4f56761bb458c8d84807feb38a617" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:37.899413 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef4f56761bb458c8d84807feb38a617" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:37.899662 27652 raft_consensus.cc:2454] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:37.899681 27258 raft_consensus.cc:2454] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:37.900050 27456 leader_election.cc:302] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:37.900300 28929 raft_consensus.cc:2781] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:37.900383 28929 raft_consensus.cc:482] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:37.900483 28929 raft_consensus.cc:3037] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.902205 28929 raft_consensus.cc:504] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:37.902647 28929 leader_election.cc:288] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:37.902976 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef4f56761bb458c8d84807feb38a617" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:37.902987 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef4f56761bb458c8d84807feb38a617" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:37.903107 27652 raft_consensus.cc:3037] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.903156 27258 raft_consensus.cc:3037] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.904847 27652 raft_consensus.cc:2454] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:37.904851 27258 raft_consensus.cc:2454] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:37.905217 27455 leader_election.cc:302] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:37.905469 28929 raft_consensus.cc:2781] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:37.905642 28929 raft_consensus.cc:686] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:17:37.905862 28929 consensus_queue.cc:237] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:37.908623 27112 catalog_manager.cc:5484] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:37.988294 29012 raft_consensus.cc:482] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:37.988472 29012 raft_consensus.cc:504] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:37.989079 29012 leader_election.cc:288] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:37.989449 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad036d297bd45d7b7c5a1f19356906f" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:37.989527 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad036d297bd45d7b7c5a1f19356906f" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:37.989686 27390 raft_consensus.cc:2454] T cad036d297bd45d7b7c5a1f19356906f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:37.989795 27520 raft_consensus.cc:2454] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:37.990180 27586 leader_election.cc:302] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:37.990475 29012 raft_consensus.cc:2781] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:37.990566 29012 raft_consensus.cc:482] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:37.990669 29012 raft_consensus.cc:3037] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.992430 29012 raft_consensus.cc:504] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:37.992955 29012 leader_election.cc:288] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:37.993239 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad036d297bd45d7b7c5a1f19356906f" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:37.993454 27390 raft_consensus.cc:3037] T cad036d297bd45d7b7c5a1f19356906f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.993415 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad036d297bd45d7b7c5a1f19356906f" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:37.993628 27520 raft_consensus.cc:3037] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.996408 27520 raft_consensus.cc:2454] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:37.996572 27390 raft_consensus.cc:2454] T cad036d297bd45d7b7c5a1f19356906f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:37.996783 27587 leader_election.cc:302] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:37.997042 29012 raft_consensus.cc:2781] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:37.997495 29012 raft_consensus.cc:686] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:17:37.997802 29012 consensus_queue.cc:237] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:38.001762 27112 catalog_manager.cc:5484] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:38.264055 29190 consensus_queue.cc:1035] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:38.271653 29190 consensus_queue.cc:1035] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:38.386354 29012 consensus_queue.cc:1035] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:38.391597 29012 consensus_queue.cc:1035] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:38.409104 29222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:38.409422 29222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:38.416440 29222 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240502 08:17:38.514943 28929 consensus_queue.cc:1035] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:38.541383 29257 consensus_queue.cc:1035] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 250.649 ms
time per row: 0.125325 ms
I20240502 08:17:38.811230 27112 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:38.818897 27237 tablet_service.cc:1467] Processing CreateTablet for tablet b57e9fcac6354a2d9d52dbd299241bd4 (DEFAULT_TABLE table=rebalance_test_table_12 [id=d1443c13acb24eacba47f4561d2d0443]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:38.819389 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b57e9fcac6354a2d9d52dbd299241bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:38.819831 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 2643aca45d404250b6d56ca12317290f (DEFAULT_TABLE table=rebalance_test_table_12 [id=d1443c13acb24eacba47f4561d2d0443]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:38.820225 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2643aca45d404250b6d56ca12317290f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:38.820207 27499 tablet_service.cc:1467] Processing CreateTablet for tablet b57e9fcac6354a2d9d52dbd299241bd4 (DEFAULT_TABLE table=rebalance_test_table_12 [id=d1443c13acb24eacba47f4561d2d0443]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:38.820143 27630 tablet_service.cc:1467] Processing CreateTablet for tablet 1c75d776e2b34e369111d7803d1b3491 (DEFAULT_TABLE table=rebalance_test_table_12 [id=d1443c13acb24eacba47f4561d2d0443]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:38.820606 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b57e9fcac6354a2d9d52dbd299241bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:38.820621 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c75d776e2b34e369111d7803d1b3491. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:38.822435 27370 tablet_service.cc:1467] Processing CreateTablet for tablet b57e9fcac6354a2d9d52dbd299241bd4 (DEFAULT_TABLE table=rebalance_test_table_12 [id=d1443c13acb24eacba47f4561d2d0443]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:38.822839 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b57e9fcac6354a2d9d52dbd299241bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:38.824671 27369 tablet_service.cc:1467] Processing CreateTablet for tablet 1c75d776e2b34e369111d7803d1b3491 (DEFAULT_TABLE table=rebalance_test_table_12 [id=d1443c13acb24eacba47f4561d2d0443]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:38.825001 27369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c75d776e2b34e369111d7803d1b3491. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:38.825667 27498 tablet_service.cc:1467] Processing CreateTablet for tablet 1c75d776e2b34e369111d7803d1b3491 (DEFAULT_TABLE table=rebalance_test_table_12 [id=d1443c13acb24eacba47f4561d2d0443]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:38.825879 27236 tablet_service.cc:1467] Processing CreateTablet for tablet 2643aca45d404250b6d56ca12317290f (DEFAULT_TABLE table=rebalance_test_table_12 [id=d1443c13acb24eacba47f4561d2d0443]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:38.826079 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c75d776e2b34e369111d7803d1b3491. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:38.826107 27368 tablet_service.cc:1467] Processing CreateTablet for tablet 2643aca45d404250b6d56ca12317290f (DEFAULT_TABLE table=rebalance_test_table_12 [id=d1443c13acb24eacba47f4561d2d0443]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:38.826247 27236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2643aca45d404250b6d56ca12317290f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:38.826467 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2643aca45d404250b6d56ca12317290f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:38.836941 29296 tablet_bootstrap.cc:492] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:38.840054 29295 tablet_bootstrap.cc:492] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:38.840415 29296 tablet_bootstrap.cc:654] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:38.842051 29295 tablet_bootstrap.cc:654] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:38.843204 29297 tablet_bootstrap.cc:492] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:38.845132 29297 tablet_bootstrap.cc:654] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:38.846185 29293 tablet_bootstrap.cc:492] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:38.847649 29295 tablet_bootstrap.cc:492] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:38.847878 29295 ts_tablet_manager.cc:1367] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20240502 08:17:38.848904 29295 raft_consensus.cc:348] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:38.849334 29295 raft_consensus.cc:374] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:38.849493 29295 raft_consensus.cc:729] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:38.849937 29295 consensus_queue.cc:260] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:38.850741 29295 ts_tablet_manager.cc:1398] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:38.851298 29295 tablet_bootstrap.cc:492] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:38.853245 29295 tablet_bootstrap.cc:654] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:38.861498 29295 tablet_bootstrap.cc:492] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:38.861689 29295 ts_tablet_manager.cc:1367] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20240502 08:17:38.861910 29293 tablet_bootstrap.cc:654] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:38.862195 29296 tablet_bootstrap.cc:492] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:38.862414 29296 ts_tablet_manager.cc:1367] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.017s
I20240502 08:17:38.862469 29295 raft_consensus.cc:348] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:38.862699 29295 raft_consensus.cc:374] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:38.862805 29295 raft_consensus.cc:729] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:38.863103 29295 consensus_queue.cc:260] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:38.863752 29295 ts_tablet_manager.cc:1398] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:38.864538 29297 tablet_bootstrap.cc:492] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:38.864686 29297 ts_tablet_manager.cc:1367] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.004s
I20240502 08:17:38.865463 29297 raft_consensus.cc:348] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:38.865671 29297 raft_consensus.cc:374] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:38.865809 29297 raft_consensus.cc:729] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:38.866061 29297 consensus_queue.cc:260] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:38.863353 29296 raft_consensus.cc:348] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:38.866299 29296 raft_consensus.cc:374] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:38.866464 29296 raft_consensus.cc:729] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:38.866708 29296 consensus_queue.cc:260] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:38.866966 29297 ts_tablet_manager.cc:1398] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:38.867507 29296 ts_tablet_manager.cc:1398] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:17:38.867946 29296 tablet_bootstrap.cc:492] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:38.868269 29297 tablet_bootstrap.cc:492] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:38.868422 29293 tablet_bootstrap.cc:492] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:38.868608 29293 ts_tablet_manager.cc:1367] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.002s
I20240502 08:17:38.869414 29293 raft_consensus.cc:348] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:38.869659 29293 raft_consensus.cc:374] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:38.869772 29296 tablet_bootstrap.cc:654] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:38.869799 29293 raft_consensus.cc:729] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:38.869843 29297 tablet_bootstrap.cc:654] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:38.870158 29293 consensus_queue.cc:260] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:38.872018 29293 ts_tablet_manager.cc:1398] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:38.872386 29293 tablet_bootstrap.cc:492] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:38.874338 29293 tablet_bootstrap.cc:654] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:38.889065 29296 tablet_bootstrap.cc:492] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:38.889221 29296 ts_tablet_manager.cc:1367] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.012s
I20240502 08:17:38.889353 29297 tablet_bootstrap.cc:492] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:38.889554 29297 ts_tablet_manager.cc:1367] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.005s
I20240502 08:17:38.889955 29296 raft_consensus.cc:348] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:38.890161 29293 tablet_bootstrap.cc:492] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:38.890245 29296 raft_consensus.cc:374] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:38.890336 29296 raft_consensus.cc:729] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:38.890424 29293 ts_tablet_manager.cc:1367] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.005s
I20240502 08:17:38.890393 29297 raft_consensus.cc:348] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:38.890625 29297 raft_consensus.cc:374] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:38.890580 29296 consensus_queue.cc:260] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:38.890700 29297 raft_consensus.cc:729] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:38.891115 29296 ts_tablet_manager.cc:1398] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:38.891230 29293 raft_consensus.cc:348] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:38.891639 29293 raft_consensus.cc:374] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:38.891687 29293 raft_consensus.cc:729] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:38.891608 29297 consensus_queue.cc:260] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:38.892145 29297 ts_tablet_manager.cc:1398] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:38.891875 29293 consensus_queue.cc:260] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:38.892864 29293 ts_tablet_manager.cc:1398] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:38.893126 29293 tablet_bootstrap.cc:492] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:38.894724 29293 tablet_bootstrap.cc:654] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:38.897933 29293 tablet_bootstrap.cc:492] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:38.898082 29293 ts_tablet_manager.cc:1367] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20240502 08:17:38.898628 29293 raft_consensus.cc:348] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:38.898794 29293 raft_consensus.cc:374] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:38.898859 29293 raft_consensus.cc:729] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:38.899004 29293 consensus_queue.cc:260] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:38.899348 29293 ts_tablet_manager.cc:1398] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:39.083305 29228 raft_consensus.cc:482] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:39.083499 29228 raft_consensus.cc:504] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:39.084296 29228 leader_election.cc:288] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:39.084703 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2643aca45d404250b6d56ca12317290f" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:39.084776 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2643aca45d404250b6d56ca12317290f" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:39.084947 27651 raft_consensus.cc:2454] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:39.084985 27258 raft_consensus.cc:2454] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:39.085450 27325 leader_election.cc:302] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:39.085920 29228 raft_consensus.cc:2781] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:39.086068 29228 raft_consensus.cc:482] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:39.086180 29228 raft_consensus.cc:3037] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:39.088057 29228 raft_consensus.cc:504] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:39.088562 29228 leader_election.cc:288] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:39.088943 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2643aca45d404250b6d56ca12317290f" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:39.088953 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2643aca45d404250b6d56ca12317290f" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:39.089095 27651 raft_consensus.cc:3037] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:39.089130 27258 raft_consensus.cc:3037] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:39.090924 27651 raft_consensus.cc:2454] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:39.091097 27258 raft_consensus.cc:2454] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:39.091271 27326 leader_election.cc:302] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:39.091526 29228 raft_consensus.cc:2781] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:39.091660 29228 raft_consensus.cc:686] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:17:39.091904 29228 consensus_queue.cc:237] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:39.094969 27116 catalog_manager.cc:5484] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:39.099269 29228 raft_consensus.cc:482] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:39.099428 29228 raft_consensus.cc:504] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:39.099977 29228 leader_election.cc:288] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:39.100301 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c75d776e2b34e369111d7803d1b3491" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:39.100509 27651 raft_consensus.cc:2454] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:39.100617 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c75d776e2b34e369111d7803d1b3491" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:39.100826 27521 raft_consensus.cc:2454] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:39.101179 27326 leader_election.cc:302] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:39.101424 29228 raft_consensus.cc:2781] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:39.101505 29228 raft_consensus.cc:482] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:39.101603 29228 raft_consensus.cc:3037] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:39.103327 29228 raft_consensus.cc:504] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:39.103770 29228 leader_election.cc:288] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:39.104115 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c75d776e2b34e369111d7803d1b3491" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:39.104174 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c75d776e2b34e369111d7803d1b3491" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:39.104326 27521 raft_consensus.cc:3037] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:39.104318 27651 raft_consensus.cc:3037] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:39.106997 27651 raft_consensus.cc:2454] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:39.107017 27521 raft_consensus.cc:2454] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:39.107342 27326 leader_election.cc:302] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:39.107559 29228 raft_consensus.cc:2781] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:39.107695 29228 raft_consensus.cc:686] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:17:39.107910 29228 consensus_queue.cc:237] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:39.110723 27116 catalog_manager.cc:5484] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:39.161084 29228 raft_consensus.cc:482] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:39.161294 29228 raft_consensus.cc:504] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:39.161864 29228 leader_election.cc:288] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:39.162221 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57e9fcac6354a2d9d52dbd299241bd4" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:39.162230 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57e9fcac6354a2d9d52dbd299241bd4" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:39.162437 27258 raft_consensus.cc:2454] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:39.162423 27521 raft_consensus.cc:2454] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:39.162766 27325 leader_election.cc:302] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:39.163039 29228 raft_consensus.cc:2781] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:39.163147 29228 raft_consensus.cc:482] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:39.163235 29228 raft_consensus.cc:3037] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:39.165117 29228 raft_consensus.cc:504] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:39.165675 29228 leader_election.cc:288] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:39.165980 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57e9fcac6354a2d9d52dbd299241bd4" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:39.166108 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57e9fcac6354a2d9d52dbd299241bd4" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:39.166215 27258 raft_consensus.cc:3037] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:39.166296 27521 raft_consensus.cc:3037] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:39.167977 27521 raft_consensus.cc:2454] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:39.167987 27258 raft_consensus.cc:2454] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:39.168315 27325 leader_election.cc:302] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:39.168586 29228 raft_consensus.cc:2781] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:39.168710 29228 raft_consensus.cc:686] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:17:39.168948 29228 consensus_queue.cc:237] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:39.171854 27116 catalog_manager.cc:5484] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:39.473888 29228 consensus_queue.cc:1035] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:39.482586 29228 consensus_queue.cc:1035] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:39.535385 29228 consensus_queue.cc:1035] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:39.540802 29228 consensus_queue.cc:1035] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:39.542095 29310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:39.542467 29310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:39.551661 29310 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:17:39.565403 29225 consensus_queue.cc:1035] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:39.572820 29228 consensus_queue.cc:1035] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_12'
INSERT report
rows total: 2000
time total: 198.779 ms
time per row: 0.0993894 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:17:40.851197 27116 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:40.859930 27236 tablet_service.cc:1467] Processing CreateTablet for tablet 903502e57ab2430383807b0d46a39847 (DEFAULT_TABLE table=rebalance_test_table_13 [id=28a67fad5bff471981e8cdf2b2a2b34a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:40.860378 27236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903502e57ab2430383807b0d46a39847. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:40.860553 27237 tablet_service.cc:1467] Processing CreateTablet for tablet 9725e08a6c5d4410b8d944ab154ec1eb (DEFAULT_TABLE table=rebalance_test_table_13 [id=28a67fad5bff471981e8cdf2b2a2b34a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:40.860734 27499 tablet_service.cc:1467] Processing CreateTablet for tablet 9725e08a6c5d4410b8d944ab154ec1eb (DEFAULT_TABLE table=rebalance_test_table_13 [id=28a67fad5bff471981e8cdf2b2a2b34a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:40.860980 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9725e08a6c5d4410b8d944ab154ec1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:40.861125 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9725e08a6c5d4410b8d944ab154ec1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:40.864997 29379 tablet_bootstrap.cc:492] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:40.865139 27498 tablet_service.cc:1467] Processing CreateTablet for tablet 903502e57ab2430383807b0d46a39847 (DEFAULT_TABLE table=rebalance_test_table_13 [id=28a67fad5bff471981e8cdf2b2a2b34a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:40.865507 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903502e57ab2430383807b0d46a39847. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:40.866849 29379 tablet_bootstrap.cc:654] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:40.866878 27369 tablet_service.cc:1467] Processing CreateTablet for tablet db55b42c724246da8d9cb23369826279 (DEFAULT_TABLE table=rebalance_test_table_13 [id=28a67fad5bff471981e8cdf2b2a2b34a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:40.867295 27369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db55b42c724246da8d9cb23369826279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:40.868744 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 9725e08a6c5d4410b8d944ab154ec1eb (DEFAULT_TABLE table=rebalance_test_table_13 [id=28a67fad5bff471981e8cdf2b2a2b34a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:40.869149 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9725e08a6c5d4410b8d944ab154ec1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:40.868726 27630 tablet_service.cc:1467] Processing CreateTablet for tablet 903502e57ab2430383807b0d46a39847 (DEFAULT_TABLE table=rebalance_test_table_13 [id=28a67fad5bff471981e8cdf2b2a2b34a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:40.870290 29379 tablet_bootstrap.cc:492] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:40.870414 29379 ts_tablet_manager.cc:1367] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:17:40.870476 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903502e57ab2430383807b0d46a39847. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:40.871059 29379 raft_consensus.cc:348] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:40.871243 29379 raft_consensus.cc:374] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:40.871316 29379 raft_consensus.cc:729] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:40.871505 29379 consensus_queue.cc:260] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:40.872489 27500 tablet_service.cc:1467] Processing CreateTablet for tablet db55b42c724246da8d9cb23369826279 (DEFAULT_TABLE table=rebalance_test_table_13 [id=28a67fad5bff471981e8cdf2b2a2b34a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:40.872707 27236 tablet_service.cc:1467] Processing CreateTablet for tablet db55b42c724246da8d9cb23369826279 (DEFAULT_TABLE table=rebalance_test_table_13 [id=28a67fad5bff471981e8cdf2b2a2b34a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:40.872810 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db55b42c724246da8d9cb23369826279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:40.872946 27236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db55b42c724246da8d9cb23369826279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:40.874157 29379 ts_tablet_manager.cc:1398] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:40.874446 29379 tablet_bootstrap.cc:492] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:40.877125 29379 tablet_bootstrap.cc:654] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:40.883884 29379 tablet_bootstrap.cc:492] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:40.884099 29379 ts_tablet_manager.cc:1367] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20240502 08:17:40.885272 29379 raft_consensus.cc:348] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:40.885607 29379 raft_consensus.cc:374] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:40.885990 29379 raft_consensus.cc:729] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:40.886435 29379 consensus_queue.cc:260] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:40.887246 29379 ts_tablet_manager.cc:1398] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:40.887348 29382 tablet_bootstrap.cc:492] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:40.888242 29383 tablet_bootstrap.cc:492] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:40.889803 29382 tablet_bootstrap.cc:654] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:40.889931 29383 tablet_bootstrap.cc:654] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:40.890444 29384 tablet_bootstrap.cc:492] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:40.891084 29379 tablet_bootstrap.cc:492] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:40.892786 29384 tablet_bootstrap.cc:654] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:40.892843 29382 tablet_bootstrap.cc:492] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:40.893250 29382 ts_tablet_manager.cc:1367] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:17:40.894412 29382 raft_consensus.cc:348] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:40.894704 29379 tablet_bootstrap.cc:654] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:40.894798 29382 raft_consensus.cc:374] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:40.894992 29382 raft_consensus.cc:729] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:40.895382 29382 consensus_queue.cc:260] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:40.897117 29383 tablet_bootstrap.cc:492] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:40.897145 29382 ts_tablet_manager.cc:1398] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:40.897257 29383 ts_tablet_manager.cc:1367] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20240502 08:17:40.897536 29379 tablet_bootstrap.cc:492] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:40.897660 29379 ts_tablet_manager.cc:1367] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 08:17:40.897791 29383 raft_consensus.cc:348] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:40.897928 29383 raft_consensus.cc:374] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:40.898047 29383 raft_consensus.cc:729] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:40.898422 29379 raft_consensus.cc:348] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:40.898859 29379 raft_consensus.cc:374] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:40.898874 29383 consensus_queue.cc:260] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:40.899006 29379 raft_consensus.cc:729] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:40.899248 29379 consensus_queue.cc:260] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:40.899451 29383 ts_tablet_manager.cc:1398] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:40.899762 29383 tablet_bootstrap.cc:492] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:40.900009 29379 ts_tablet_manager.cc:1398] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:40.902544 29383 tablet_bootstrap.cc:654] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:40.904202 29384 tablet_bootstrap.cc:492] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:40.904304 29383 tablet_bootstrap.cc:492] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:40.904347 29384 ts_tablet_manager.cc:1367] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240502 08:17:40.904436 29383 ts_tablet_manager.cc:1367] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:17:40.904893 29384 raft_consensus.cc:348] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:40.904901 29383 raft_consensus.cc:348] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:40.905043 29383 raft_consensus.cc:374] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:40.905040 29384 raft_consensus.cc:374] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:40.905112 29383 raft_consensus.cc:729] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:40.905134 29384 raft_consensus.cc:729] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:40.905334 29383 consensus_queue.cc:260] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:40.905334 29384 consensus_queue.cc:260] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:40.905732 29384 ts_tablet_manager.cc:1398] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:40.905781 29383 ts_tablet_manager.cc:1398] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 08:17:40.906113 29383 tablet_bootstrap.cc:492] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:40.906579 29384 tablet_bootstrap.cc:492] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:40.907552 29383 tablet_bootstrap.cc:654] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:40.909126 29384 tablet_bootstrap.cc:654] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:40.911914 29384 tablet_bootstrap.cc:492] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:40.912034 29384 ts_tablet_manager.cc:1367] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:17:40.912506 29384 raft_consensus.cc:348] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:40.912694 29384 raft_consensus.cc:374] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:40.912768 29384 raft_consensus.cc:729] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:40.912976 29384 consensus_queue.cc:260] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:40.913417 29384 ts_tablet_manager.cc:1398] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:40.915205 29383 tablet_bootstrap.cc:492] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:40.915330 29383 ts_tablet_manager.cc:1367] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20240502 08:17:40.915935 29383 raft_consensus.cc:348] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:40.916124 29383 raft_consensus.cc:374] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:40.916217 29383 raft_consensus.cc:729] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:40.916419 29383 consensus_queue.cc:260] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:40.916860 29383 ts_tablet_manager.cc:1398] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:40.961681 28929 raft_consensus.cc:482] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:40.961863 28929 raft_consensus.cc:504] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:40.962582 28929 leader_election.cc:288] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:40.962909 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9725e08a6c5d4410b8d944ab154ec1eb" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:40.963001 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9725e08a6c5d4410b8d944ab154ec1eb" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:40.963183 27651 raft_consensus.cc:2454] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:40.963189 27258 raft_consensus.cc:2454] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:40.963558 27456 leader_election.cc:302] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:40.963783 28929 raft_consensus.cc:2781] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:40.963866 28929 raft_consensus.cc:482] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:40.963980 28929 raft_consensus.cc:3037] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:40.967272 28929 raft_consensus.cc:504] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:40.968952 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9725e08a6c5d4410b8d944ab154ec1eb" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:40.969373 27258 raft_consensus.cc:3037] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:40.969516 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9725e08a6c5d4410b8d944ab154ec1eb" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:40.970067 27651 raft_consensus.cc:3037] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:40.970595 28929 leader_election.cc:288] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:40.972496 27258 raft_consensus.cc:2454] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:40.972836 27455 leader_election.cc:302] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:40.972932 27651 raft_consensus.cc:2454] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:40.973083 28929 raft_consensus.cc:2781] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:40.973256 28929 raft_consensus.cc:686] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:17:40.973520 28929 consensus_queue.cc:237] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:40.976526 27111 catalog_manager.cc:5484] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:41.031951 28929 raft_consensus.cc:482] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:41.032120 28929 raft_consensus.cc:504] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:41.032680 28929 leader_election.cc:288] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:41.032990 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903502e57ab2430383807b0d46a39847" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:41.033041 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903502e57ab2430383807b0d46a39847" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:41.033197 27258 raft_consensus.cc:2454] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:41.033273 27651 raft_consensus.cc:2454] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:41.033529 27455 leader_election.cc:302] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:41.033751 28929 raft_consensus.cc:2781] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:41.033854 28929 raft_consensus.cc:482] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:41.033954 28929 raft_consensus.cc:3037] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:41.034029 29392 raft_consensus.cc:482] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:41.034150 29392 raft_consensus.cc:504] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:41.034847 29392 leader_election.cc:288] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:41.035120 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55b42c724246da8d9cb23369826279" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:41.035379 27390 raft_consensus.cc:2454] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:41.035332 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55b42c724246da8d9cb23369826279" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:41.035560 27258 raft_consensus.cc:2454] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:41.035966 27455 leader_election.cc:302] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:41.036211 29392 raft_consensus.cc:2781] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:41.036298 29392 raft_consensus.cc:482] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:41.036371 29392 raft_consensus.cc:3037] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:41.036633 28929 raft_consensus.cc:504] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:41.037222 28929 leader_election.cc:288] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:41.037483 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903502e57ab2430383807b0d46a39847" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:41.037578 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903502e57ab2430383807b0d46a39847" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:41.037644 27258 raft_consensus.cc:3037] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:41.037775 27651 raft_consensus.cc:3037] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:41.038774 29392 raft_consensus.cc:504] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:41.039324 29392 leader_election.cc:288] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:41.039669 27258 raft_consensus.cc:2454] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:41.039638 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55b42c724246da8d9cb23369826279" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:41.039834 27390 raft_consensus.cc:3037] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:41.040050 27455 leader_election.cc:302] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:41.040196 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db55b42c724246da8d9cb23369826279" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:41.040339 27258 raft_consensus.cc:3037] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:41.040707 27651 raft_consensus.cc:2454] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:41.041990 27390 raft_consensus.cc:2454] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:41.042052 27258 raft_consensus.cc:2454] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:41.042351 29392 raft_consensus.cc:2781] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:41.042474 29392 raft_consensus.cc:686] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:17:41.042423 27455 leader_election.cc:302] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:41.042711 29392 consensus_queue.cc:237] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:41.042971 28929 raft_consensus.cc:2781] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:41.043430 28929 raft_consensus.cc:686] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:17:41.043741 28929 consensus_queue.cc:237] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:41.046002 27111 catalog_manager.cc:5484] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:41.050351 27111 catalog_manager.cc:5484] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:41.395587 29397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:41.395876 29397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:41.403503 29397 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240502 08:17:41.484434 28929 consensus_queue.cc:1035] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:41.501441 29394 consensus_queue.cc:1035] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:41.529392 29394 consensus_queue.cc:1035] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:41.552256 28929 consensus_queue.cc:1035] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:41.575271 29392 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:41.581949 29392 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 312.801 ms
time per row: 0.1564 ms
I20240502 08:17:41.865144 27111 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:41.873389 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 9935a8efe788429dbdfc8c61387adb60 (DEFAULT_TABLE table=rebalance_test_table_14 [id=ecb31f1a638e457fa80fdec831562539]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:41.873625 27236 tablet_service.cc:1467] Processing CreateTablet for tablet 9935a8efe788429dbdfc8c61387adb60 (DEFAULT_TABLE table=rebalance_test_table_14 [id=ecb31f1a638e457fa80fdec831562539]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:41.873811 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9935a8efe788429dbdfc8c61387adb60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:41.874001 27236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9935a8efe788429dbdfc8c61387adb60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:41.874850 27370 tablet_service.cc:1467] Processing CreateTablet for tablet b613c84f191b4dfa9b306fa605456b95 (DEFAULT_TABLE table=rebalance_test_table_14 [id=ecb31f1a638e457fa80fdec831562539]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:41.874887 27499 tablet_service.cc:1467] Processing CreateTablet for tablet b613c84f191b4dfa9b306fa605456b95 (DEFAULT_TABLE table=rebalance_test_table_14 [id=ecb31f1a638e457fa80fdec831562539]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:41.875206 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b613c84f191b4dfa9b306fa605456b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:41.875306 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b613c84f191b4dfa9b306fa605456b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:41.878494 27369 tablet_service.cc:1467] Processing CreateTablet for tablet 9935a8efe788429dbdfc8c61387adb60 (DEFAULT_TABLE table=rebalance_test_table_14 [id=ecb31f1a638e457fa80fdec831562539]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:41.878894 27369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9935a8efe788429dbdfc8c61387adb60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:41.879086 27630 tablet_service.cc:1467] Processing CreateTablet for tablet b613c84f191b4dfa9b306fa605456b95 (DEFAULT_TABLE table=rebalance_test_table_14 [id=ecb31f1a638e457fa80fdec831562539]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:41.879456 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b613c84f191b4dfa9b306fa605456b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:41.880239 29458 tablet_bootstrap.cc:492] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:41.881573 29458 tablet_bootstrap.cc:654] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:41.882421 27629 tablet_service.cc:1467] Processing CreateTablet for tablet dd7732b2d14c4072b64fa290f1c613ab (DEFAULT_TABLE table=rebalance_test_table_14 [id=ecb31f1a638e457fa80fdec831562539]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:41.882751 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7732b2d14c4072b64fa290f1c613ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:41.883215 27370 tablet_service.cc:1467] Processing CreateTablet for tablet dd7732b2d14c4072b64fa290f1c613ab (DEFAULT_TABLE table=rebalance_test_table_14 [id=ecb31f1a638e457fa80fdec831562539]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:41.883675 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7732b2d14c4072b64fa290f1c613ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:41.886750 27237 tablet_service.cc:1467] Processing CreateTablet for tablet dd7732b2d14c4072b64fa290f1c613ab (DEFAULT_TABLE table=rebalance_test_table_14 [id=ecb31f1a638e457fa80fdec831562539]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:41.887151 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7732b2d14c4072b64fa290f1c613ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:41.899785 29460 tablet_bootstrap.cc:492] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:41.901501 29460 tablet_bootstrap.cc:654] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:41.903393 29461 tablet_bootstrap.cc:492] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:41.905508 29461 tablet_bootstrap.cc:654] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:41.906185 29462 tablet_bootstrap.cc:492] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:41.908241 29458 tablet_bootstrap.cc:492] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:41.908282 29462 tablet_bootstrap.cc:654] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:41.908450 29458 ts_tablet_manager.cc:1367] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.009s
I20240502 08:17:41.909299 29458 raft_consensus.cc:348] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:41.909610 29458 raft_consensus.cc:374] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:41.909746 29458 raft_consensus.cc:729] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:41.909963 29458 consensus_queue.cc:260] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:41.910259 29461 tablet_bootstrap.cc:492] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:41.910526 29461 ts_tablet_manager.cc:1367] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 08:17:41.910712 29458 ts_tablet_manager.cc:1398] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:41.911047 29458 tablet_bootstrap.cc:492] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:41.911264 29461 raft_consensus.cc:348] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:41.911476 29461 raft_consensus.cc:374] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:41.911558 29461 raft_consensus.cc:729] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:41.911796 29461 consensus_queue.cc:260] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:41.912380 29461 ts_tablet_manager.cc:1398] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:41.913126 29458 tablet_bootstrap.cc:654] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:41.917873 29462 tablet_bootstrap.cc:492] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:41.918304 29462 ts_tablet_manager.cc:1367] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.003s
I20240502 08:17:41.919060 29462 raft_consensus.cc:348] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:41.919255 29462 raft_consensus.cc:374] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:41.919348 29462 raft_consensus.cc:729] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:41.919584 29462 consensus_queue.cc:260] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:41.920152 29462 ts_tablet_manager.cc:1398] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:41.922065 29462 tablet_bootstrap.cc:492] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:41.923213 29458 tablet_bootstrap.cc:492] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:41.923382 29458 ts_tablet_manager.cc:1367] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240502 08:17:41.924082 29458 raft_consensus.cc:348] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:41.924360 29458 raft_consensus.cc:374] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:41.924479 29458 raft_consensus.cc:729] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:41.924783 29458 consensus_queue.cc:260] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:41.925351 29458 ts_tablet_manager.cc:1398] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:41.925722 29458 tablet_bootstrap.cc:492] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:41.926501 29460 tablet_bootstrap.cc:492] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:41.926740 29460 ts_tablet_manager.cc:1367] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.000s
I20240502 08:17:41.927469 29458 tablet_bootstrap.cc:654] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:41.927505 29460 raft_consensus.cc:348] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:41.927696 29460 raft_consensus.cc:374] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:41.927791 29460 raft_consensus.cc:729] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:41.928005 29460 consensus_queue.cc:260] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:41.928611 29460 ts_tablet_manager.cc:1398] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:41.929806 29460 tablet_bootstrap.cc:492] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:41.933456 29460 tablet_bootstrap.cc:654] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:41.939313 29462 tablet_bootstrap.cc:654] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:41.941627 29462 tablet_bootstrap.cc:492] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:41.941642 29458 tablet_bootstrap.cc:492] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:41.941776 29458 ts_tablet_manager.cc:1367] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20240502 08:17:41.941793 29462 ts_tablet_manager.cc:1367] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240502 08:17:41.942593 29458 raft_consensus.cc:348] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:41.942584 29462 raft_consensus.cc:348] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:41.942860 29458 raft_consensus.cc:374] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:41.942884 29460 tablet_bootstrap.cc:492] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:41.942987 29462 raft_consensus.cc:374] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:41.943009 29458 raft_consensus.cc:729] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:41.943136 29462 raft_consensus.cc:729] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:41.943142 29460 ts_tablet_manager.cc:1367] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240502 08:17:41.943351 29458 consensus_queue.cc:260] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:41.943518 29462 consensus_queue.cc:260] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:41.944105 29458 ts_tablet_manager.cc:1398] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:41.944093 29460 raft_consensus.cc:348] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:41.944370 29460 raft_consensus.cc:374] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:41.944455 29460 raft_consensus.cc:729] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:41.944849 29460 consensus_queue.cc:260] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:41.944972 29462 ts_tablet_manager.cc:1398] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:17:41.945449 29460 ts_tablet_manager.cc:1398] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:41.945796 29460 tablet_bootstrap.cc:492] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:41.947582 29460 tablet_bootstrap.cc:654] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:41.949395 29460 tablet_bootstrap.cc:492] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:41.949517 29460 ts_tablet_manager.cc:1367] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 08:17:41.950368 29460 raft_consensus.cc:348] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:41.950522 29460 raft_consensus.cc:374] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:41.950608 29460 raft_consensus.cc:729] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:41.950798 29460 consensus_queue.cc:260] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:41.951253 29460 ts_tablet_manager.cc:1398] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:42.011551 29449 raft_consensus.cc:482] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:42.011708 29449 raft_consensus.cc:504] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:42.012573 29449 leader_election.cc:288] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:42.012970 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7732b2d14c4072b64fa290f1c613ab" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:42.012970 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7732b2d14c4072b64fa290f1c613ab" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:42.013211 27652 raft_consensus.cc:2454] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:42.013192 27387 raft_consensus.cc:2454] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:42.013569 27193 leader_election.cc:302] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:42.013814 29449 raft_consensus.cc:2781] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:42.013922 29449 raft_consensus.cc:482] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:42.014017 29449 raft_consensus.cc:3037] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:42.016160 29449 raft_consensus.cc:504] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:42.016669 29449 leader_election.cc:288] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:42.017007 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7732b2d14c4072b64fa290f1c613ab" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:42.017113 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7732b2d14c4072b64fa290f1c613ab" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:42.017263 27652 raft_consensus.cc:3037] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:42.017300 27387 raft_consensus.cc:3037] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:42.020045 27387 raft_consensus.cc:2454] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:42.020030 27652 raft_consensus.cc:2454] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:42.020412 27194 leader_election.cc:302] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:42.020699 29449 raft_consensus.cc:2781] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:42.020813 29449 raft_consensus.cc:686] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:42.021056 29449 consensus_queue.cc:237] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:42.024046 27113 catalog_manager.cc:5484] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:42.052253 29392 raft_consensus.cc:482] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:42.052459 29392 raft_consensus.cc:504] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:42.053025 29392 leader_election.cc:288] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:42.053409 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b613c84f191b4dfa9b306fa605456b95" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:42.053463 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b613c84f191b4dfa9b306fa605456b95" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:42.053644 27652 raft_consensus.cc:2454] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:42.053647 27387 raft_consensus.cc:2454] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:42.054044 27456 leader_election.cc:302] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:42.054323 29392 raft_consensus.cc:2781] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:42.054419 29392 raft_consensus.cc:482] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:42.054520 29392 raft_consensus.cc:3037] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:42.056937 29392 raft_consensus.cc:504] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:42.057480 29392 leader_election.cc:288] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:42.057811 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b613c84f191b4dfa9b306fa605456b95" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:42.057873 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b613c84f191b4dfa9b306fa605456b95" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:42.057976 27387 raft_consensus.cc:3037] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:42.058075 27652 raft_consensus.cc:3037] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:42.059795 27387 raft_consensus.cc:2454] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:42.060173 27455 leader_election.cc:302] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:42.060405 29392 raft_consensus.cc:2781] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:42.060385 27652 raft_consensus.cc:2454] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:42.060570 29392 raft_consensus.cc:686] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:17:42.060828 29392 consensus_queue.cc:237] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:42.063920 27113 catalog_manager.cc:5484] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:42.108197 29449 raft_consensus.cc:482] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:42.108366 29449 raft_consensus.cc:504] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:42.108939 29449 leader_election.cc:288] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:42.109280 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9935a8efe788429dbdfc8c61387adb60" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:42.109465 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9935a8efe788429dbdfc8c61387adb60" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:42.109544 27652 raft_consensus.cc:2454] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:42.109719 27387 raft_consensus.cc:2454] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:42.109941 27194 leader_election.cc:302] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:42.110320 29449 raft_consensus.cc:2781] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:42.110396 29449 raft_consensus.cc:482] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:42.110483 29449 raft_consensus.cc:3037] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:42.112250 29449 raft_consensus.cc:504] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:42.112709 29449 leader_election.cc:288] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:42.112957 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9935a8efe788429dbdfc8c61387adb60" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:42.113200 27652 raft_consensus.cc:3037] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:42.113166 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9935a8efe788429dbdfc8c61387adb60" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:42.113387 27387 raft_consensus.cc:3037] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:42.115952 27652 raft_consensus.cc:2454] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:42.116079 27387 raft_consensus.cc:2454] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:42.116343 27194 leader_election.cc:302] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:42.116602 29449 raft_consensus.cc:2781] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:42.116745 29449 raft_consensus.cc:686] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:42.116986 29449 consensus_queue.cc:237] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:42.120522 27113 catalog_manager.cc:5484] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:42.452360 29392 consensus_queue.cc:1035] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:42.464461 29394 consensus_queue.cc:1035] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:42.488094 29474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:42.488365 29474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:42.495857 29474 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240502 08:17:42.528759 29449 consensus_queue.cc:1035] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:42.543408 29000 consensus_queue.cc:1035] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:42.641770 27387 raft_consensus.cc:1261] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:42.642035 27650 raft_consensus.cc:1261] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:42.642400 29000 consensus_queue.cc:1035] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:42.645556 29000 consensus_queue.cc:1035] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 276.69 ms
time per row: 0.138345 ms
I20240502 08:17:42.921541 27116 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:42.929502 27499 tablet_service.cc:1467] Processing CreateTablet for tablet 1662e908146944fd8e74d901e497e35d (DEFAULT_TABLE table=rebalance_test_table_15 [id=f2de4023ae664751b94f83ff4434cd92]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:42.930039 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1662e908146944fd8e74d901e497e35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:42.930310 27236 tablet_service.cc:1467] Processing CreateTablet for tablet 1662e908146944fd8e74d901e497e35d (DEFAULT_TABLE table=rebalance_test_table_15 [id=f2de4023ae664751b94f83ff4434cd92]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:42.930485 27498 tablet_service.cc:1467] Processing CreateTablet for tablet a6dce110666a4a038f5f46d48a3ec7af (DEFAULT_TABLE table=rebalance_test_table_15 [id=f2de4023ae664751b94f83ff4434cd92]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:42.930701 27236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1662e908146944fd8e74d901e497e35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:42.930790 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6dce110666a4a038f5f46d48a3ec7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:42.931236 27369 tablet_service.cc:1467] Processing CreateTablet for tablet 1662e908146944fd8e74d901e497e35d (DEFAULT_TABLE table=rebalance_test_table_15 [id=f2de4023ae664751b94f83ff4434cd92]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:42.931602 27369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1662e908146944fd8e74d901e497e35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:42.934530 27630 tablet_service.cc:1467] Processing CreateTablet for tablet a6dce110666a4a038f5f46d48a3ec7af (DEFAULT_TABLE table=rebalance_test_table_15 [id=f2de4023ae664751b94f83ff4434cd92]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:42.934630 27370 tablet_service.cc:1467] Processing CreateTablet for tablet a6dce110666a4a038f5f46d48a3ec7af (DEFAULT_TABLE table=rebalance_test_table_15 [id=f2de4023ae664751b94f83ff4434cd92]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:42.934934 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6dce110666a4a038f5f46d48a3ec7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:42.935007 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6dce110666a4a038f5f46d48a3ec7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:42.938694 27629 tablet_service.cc:1467] Processing CreateTablet for tablet eb5345b931504cfdbb272189b719efa4 (DEFAULT_TABLE table=rebalance_test_table_15 [id=f2de4023ae664751b94f83ff4434cd92]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:42.939082 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5345b931504cfdbb272189b719efa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:42.939661 27500 tablet_service.cc:1467] Processing CreateTablet for tablet eb5345b931504cfdbb272189b719efa4 (DEFAULT_TABLE table=rebalance_test_table_15 [id=f2de4023ae664751b94f83ff4434cd92]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:42.940044 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5345b931504cfdbb272189b719efa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:42.943385 27237 tablet_service.cc:1467] Processing CreateTablet for tablet eb5345b931504cfdbb272189b719efa4 (DEFAULT_TABLE table=rebalance_test_table_15 [id=f2de4023ae664751b94f83ff4434cd92]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:42.943912 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5345b931504cfdbb272189b719efa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:42.955030 29543 tablet_bootstrap.cc:492] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:42.957780 29543 tablet_bootstrap.cc:654] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:42.963415 29545 tablet_bootstrap.cc:492] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:42.964215 29541 tablet_bootstrap.cc:492] T 1662e908146944fd8e74d901e497e35d P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:42.965644 29545 tablet_bootstrap.cc:654] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:42.966858 29541 tablet_bootstrap.cc:654] T 1662e908146944fd8e74d901e497e35d P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:42.967106 29542 tablet_bootstrap.cc:492] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:42.969413 29542 tablet_bootstrap.cc:654] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:42.971813 29543 tablet_bootstrap.cc:492] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:42.972016 29543 ts_tablet_manager.cc:1367] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20240502 08:17:42.972321 29541 tablet_bootstrap.cc:492] T 1662e908146944fd8e74d901e497e35d P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:42.972687 29541 ts_tablet_manager.cc:1367] T 1662e908146944fd8e74d901e497e35d P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240502 08:17:42.972823 29543 raft_consensus.cc:348] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:42.973138 29543 raft_consensus.cc:374] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:42.973274 29543 raft_consensus.cc:729] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:42.973636 29543 consensus_queue.cc:260] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:42.973937 29541 raft_consensus.cc:348] T 1662e908146944fd8e74d901e497e35d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:42.974198 29541 raft_consensus.cc:374] T 1662e908146944fd8e74d901e497e35d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:42.974351 29541 raft_consensus.cc:729] T 1662e908146944fd8e74d901e497e35d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:42.974642 29541 consensus_queue.cc:260] T 1662e908146944fd8e74d901e497e35d P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:42.975309 29541 ts_tablet_manager.cc:1398] T 1662e908146944fd8e74d901e497e35d P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:42.976203 29542 tablet_bootstrap.cc:492] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:42.976338 29542 ts_tablet_manager.cc:1367] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240502 08:17:42.976748 29545 tablet_bootstrap.cc:492] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:42.976922 29545 ts_tablet_manager.cc:1367] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20240502 08:17:42.976920 29542 raft_consensus.cc:348] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:42.977115 29542 raft_consensus.cc:374] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:42.977206 29542 raft_consensus.cc:729] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:42.977427 29542 consensus_queue.cc:260] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:42.977835 29541 tablet_bootstrap.cc:492] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:42.978085 29545 raft_consensus.cc:348] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:42.978353 29545 raft_consensus.cc:374] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:42.978441 29545 raft_consensus.cc:729] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:42.978546 29543 ts_tablet_manager.cc:1398] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:17:42.978750 29545 consensus_queue.cc:260] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:42.979331 29541 tablet_bootstrap.cc:654] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:42.980429 29545 ts_tablet_manager.cc:1398] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:42.980738 29545 tablet_bootstrap.cc:492] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:42.981240 29543 tablet_bootstrap.cc:492] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:42.982573 29542 ts_tablet_manager.cc:1398] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:17:42.982863 29542 tablet_bootstrap.cc:492] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:42.983129 29543 tablet_bootstrap.cc:654] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:42.984397 29542 tablet_bootstrap.cc:654] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:42.984570 29545 tablet_bootstrap.cc:654] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:42.986235 29541 tablet_bootstrap.cc:492] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:42.986430 29541 ts_tablet_manager.cc:1367] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 08:17:42.987790 29541 raft_consensus.cc:348] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:42.988049 29541 raft_consensus.cc:374] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:42.988121 29541 raft_consensus.cc:729] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:42.988338 29541 consensus_queue.cc:260] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:42.989006 29541 ts_tablet_manager.cc:1398] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:42.992125 29543 tablet_bootstrap.cc:492] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:42.992312 29543 ts_tablet_manager.cc:1367] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240502 08:17:42.992619 29542 tablet_bootstrap.cc:492] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:42.992792 29542 ts_tablet_manager.cc:1367] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 08:17:42.993513 29542 raft_consensus.cc:348] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:42.993783 29542 raft_consensus.cc:374] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:42.993917 29542 raft_consensus.cc:729] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:42.994252 29542 consensus_queue.cc:260] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:42.994485 29545 tablet_bootstrap.cc:492] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:42.994719 29545 ts_tablet_manager.cc:1367] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.007s
I20240502 08:17:42.995522 29542 ts_tablet_manager.cc:1398] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:17:42.995436 29545 raft_consensus.cc:348] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:42.995790 29542 tablet_bootstrap.cc:492] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:42.995635 29545 raft_consensus.cc:374] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:42.996058 29545 raft_consensus.cc:729] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:42.996558 29545 consensus_queue.cc:260] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:42.996681 29543 raft_consensus.cc:348] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:42.996977 29543 raft_consensus.cc:374] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:42.997100 29543 raft_consensus.cc:729] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:42.997356 29542 tablet_bootstrap.cc:654] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:42.997143 29545 ts_tablet_manager.cc:1398] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:42.997402 29543 consensus_queue.cc:260] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:42.999281 29543 ts_tablet_manager.cc:1398] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:17:43.003551 29542 tablet_bootstrap.cc:492] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:43.003677 29542 ts_tablet_manager.cc:1367] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20240502 08:17:43.004333 29542 raft_consensus.cc:348] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:43.004528 29542 raft_consensus.cc:374] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:43.004627 29542 raft_consensus.cc:729] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:43.004817 29542 consensus_queue.cc:260] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:43.005388 29542 ts_tablet_manager.cc:1398] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:43.009681 29478 raft_consensus.cc:482] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:43.009794 29478 raft_consensus.cc:504] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:43.010382 29478 leader_election.cc:288] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:43.010681 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6dce110666a4a038f5f46d48a3ec7af" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:43.010816 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6dce110666a4a038f5f46d48a3ec7af" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:43.010886 27650 raft_consensus.cc:2454] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:43.011008 27390 raft_consensus.cc:2454] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:43.011221 27456 leader_election.cc:302] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:43.011458 29478 raft_consensus.cc:2781] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:43.011546 29478 raft_consensus.cc:482] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:43.011623 29478 raft_consensus.cc:3037] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:43.013407 29478 raft_consensus.cc:504] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:43.013921 29478 leader_election.cc:288] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:43.014289 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6dce110666a4a038f5f46d48a3ec7af" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:43.014452 27650 raft_consensus.cc:3037] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:43.014477 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6dce110666a4a038f5f46d48a3ec7af" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:43.014648 27390 raft_consensus.cc:3037] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:43.016515 27650 raft_consensus.cc:2454] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:43.016825 27390 raft_consensus.cc:2454] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:43.016865 27456 leader_election.cc:302] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:43.017123 29478 raft_consensus.cc:2781] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:43.017343 29478 raft_consensus.cc:686] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:17:43.017647 29478 consensus_queue.cc:237] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:43.021068 27116 catalog_manager.cc:5484] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:43.077442 29478 raft_consensus.cc:482] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:43.077656 29478 raft_consensus.cc:504] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:43.078150 29478 leader_election.cc:288] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:43.078469 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5345b931504cfdbb272189b719efa4" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:43.078490 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5345b931504cfdbb272189b719efa4" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:43.078661 27258 raft_consensus.cc:2454] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:43.078683 27650 raft_consensus.cc:2454] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:43.079066 27456 leader_election.cc:302] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:43.079322 29478 raft_consensus.cc:2781] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:43.079427 29478 raft_consensus.cc:482] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:43.079536 29478 raft_consensus.cc:3037] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:43.081384 29478 raft_consensus.cc:504] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:43.081898 29478 leader_election.cc:288] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:43.082159 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5345b931504cfdbb272189b719efa4" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:43.082248 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5345b931504cfdbb272189b719efa4" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:43.082314 27258 raft_consensus.cc:3037] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:43.082391 27650 raft_consensus.cc:3037] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:43.083963 27258 raft_consensus.cc:2454] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:43.084084 27650 raft_consensus.cc:2454] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:43.084390 27455 leader_election.cc:302] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:43.084650 29478 raft_consensus.cc:2781] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:43.084767 29478 raft_consensus.cc:686] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:17:43.084968 29478 consensus_queue.cc:237] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:43.087693 27116 catalog_manager.cc:5484] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:43.165431 29526 raft_consensus.cc:482] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:43.165563 29526 raft_consensus.cc:504] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:43.166177 29526 leader_election.cc:288] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:43.166527 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1662e908146944fd8e74d901e497e35d" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:43.166538 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1662e908146944fd8e74d901e497e35d" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:43.166707 27521 raft_consensus.cc:2454] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:43.166718 27387 raft_consensus.cc:2454] T 1662e908146944fd8e74d901e497e35d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:43.167068 27193 leader_election.cc:302] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:43.167326 29526 raft_consensus.cc:2781] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:43.167414 29526 raft_consensus.cc:482] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:43.167483 29526 raft_consensus.cc:3037] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:43.169452 29526 raft_consensus.cc:504] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:43.169991 29526 leader_election.cc:288] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:43.170329 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1662e908146944fd8e74d901e497e35d" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:43.170457 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1662e908146944fd8e74d901e497e35d" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:43.170524 27521 raft_consensus.cc:3037] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:43.170629 27387 raft_consensus.cc:3037] T 1662e908146944fd8e74d901e497e35d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:43.173007 27521 raft_consensus.cc:2454] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:43.173089 27387 raft_consensus.cc:2454] T 1662e908146944fd8e74d901e497e35d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:43.173391 27194 leader_election.cc:302] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:43.173674 29526 raft_consensus.cc:2781] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:43.173817 29526 raft_consensus.cc:686] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:43.174321 29526 consensus_queue.cc:237] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:43.177340 27116 catalog_manager.cc:5484] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:43.395738 29478 consensus_queue.cc:1035] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:43.410621 29478 consensus_queue.cc:1035] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:43.525570 29557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:43.525861 29557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:43.533727 29557 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:17:43.550976 29562 consensus_queue.cc:1035] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:43.558108 29562 consensus_queue.cc:1035] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_15'
I20240502 08:17:43.648500 29449 consensus_queue.cc:1035] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:43.653483 29526 consensus_queue.cc:1035] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:43.780351 27194 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.4:33309, user_credentials={real_user=slave}} blocked reactor thread for 73243.3us
INSERT report
rows total: 2000
time total: 254.573 ms
time per row: 0.127287 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:17:44.855834 27110 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:44.864274 27236 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc33dff3dee4b3fbd0620b5d2faaea1 (DEFAULT_TABLE table=rebalance_test_table_16 [id=34d6118f78a74939856a95191f42c39d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:44.864557 27498 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc33dff3dee4b3fbd0620b5d2faaea1 (DEFAULT_TABLE table=rebalance_test_table_16 [id=34d6118f78a74939856a95191f42c39d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:44.864714 27236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc33dff3dee4b3fbd0620b5d2faaea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.864923 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc33dff3dee4b3fbd0620b5d2faaea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.866091 27369 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc33dff3dee4b3fbd0620b5d2faaea1 (DEFAULT_TABLE table=rebalance_test_table_16 [id=34d6118f78a74939856a95191f42c39d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:44.866278 27370 tablet_service.cc:1467] Processing CreateTablet for tablet d8c312b65213444f9e946a932d98874c (DEFAULT_TABLE table=rebalance_test_table_16 [id=34d6118f78a74939856a95191f42c39d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:44.866514 27369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc33dff3dee4b3fbd0620b5d2faaea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.866923 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c312b65213444f9e946a932d98874c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.869529 27368 tablet_service.cc:1467] Processing CreateTablet for tablet 4c7d725414664b64b0ca58babe5df1ff (DEFAULT_TABLE table=rebalance_test_table_16 [id=34d6118f78a74939856a95191f42c39d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:44.869874 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c7d725414664b64b0ca58babe5df1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.870765 27499 tablet_service.cc:1467] Processing CreateTablet for tablet 4c7d725414664b64b0ca58babe5df1ff (DEFAULT_TABLE table=rebalance_test_table_16 [id=34d6118f78a74939856a95191f42c39d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:44.871124 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c7d725414664b64b0ca58babe5df1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.874940 27237 tablet_service.cc:1467] Processing CreateTablet for tablet d8c312b65213444f9e946a932d98874c (DEFAULT_TABLE table=rebalance_test_table_16 [id=34d6118f78a74939856a95191f42c39d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:44.875397 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c312b65213444f9e946a932d98874c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.878425 29622 tablet_bootstrap.cc:492] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:44.878540 27630 tablet_service.cc:1467] Processing CreateTablet for tablet d8c312b65213444f9e946a932d98874c (DEFAULT_TABLE table=rebalance_test_table_16 [id=34d6118f78a74939856a95191f42c39d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:44.878980 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c312b65213444f9e946a932d98874c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.879204 29623 tablet_bootstrap.cc:492] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:44.879555 27629 tablet_service.cc:1467] Processing CreateTablet for tablet 4c7d725414664b64b0ca58babe5df1ff (DEFAULT_TABLE table=rebalance_test_table_16 [id=34d6118f78a74939856a95191f42c39d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:44.879900 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c7d725414664b64b0ca58babe5df1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.880082 29622 tablet_bootstrap.cc:654] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.880798 29623 tablet_bootstrap.cc:654] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.898473 29622 tablet_bootstrap.cc:492] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:44.898727 29622 ts_tablet_manager.cc:1367] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.017s
I20240502 08:17:44.898849 29627 tablet_bootstrap.cc:492] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:44.899554 29622 raft_consensus.cc:348] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:44.899966 29622 raft_consensus.cc:374] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.900136 29622 raft_consensus.cc:729] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.900424 29622 consensus_queue.cc:260] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:44.902696 29622 ts_tablet_manager.cc:1398] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:44.903458 29622 tablet_bootstrap.cc:492] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:44.906317 29627 tablet_bootstrap.cc:654] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.909507 29622 tablet_bootstrap.cc:654] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.911849 29626 tablet_bootstrap.cc:492] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:44.914150 29627 tablet_bootstrap.cc:492] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:44.914350 29627 ts_tablet_manager.cc:1367] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20240502 08:17:44.914687 29626 tablet_bootstrap.cc:654] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.915376 29627 raft_consensus.cc:348] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:44.916505 29627 raft_consensus.cc:374] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.917212 29627 raft_consensus.cc:729] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.917483 29627 consensus_queue.cc:260] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:44.919523 29627 ts_tablet_manager.cc:1398] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:17:44.919981 29627 tablet_bootstrap.cc:492] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:44.922904 29627 tablet_bootstrap.cc:654] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.922935 29623 tablet_bootstrap.cc:492] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:44.923105 29623 ts_tablet_manager.cc:1367] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.024s
I20240502 08:17:44.924351 29626 tablet_bootstrap.cc:492] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:44.924556 29626 ts_tablet_manager.cc:1367] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 08:17:44.926035 29626 raft_consensus.cc:348] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:44.926368 29626 raft_consensus.cc:374] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.926447 29622 tablet_bootstrap.cc:492] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:44.926401 29623 raft_consensus.cc:348] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:44.926605 29622 ts_tablet_manager.cc:1367] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.003s
I20240502 08:17:44.926614 29623 raft_consensus.cc:374] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.926642 29627 tablet_bootstrap.cc:492] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:44.926564 29626 raft_consensus.cc:729] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.926700 29623 raft_consensus.cc:729] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.926785 29627 ts_tablet_manager.cc:1367] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240502 08:17:44.926947 29623 consensus_queue.cc:260] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:44.927026 29626 consensus_queue.cc:260] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:44.927328 29622 raft_consensus.cc:348] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:44.927565 29622 raft_consensus.cc:374] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.927628 29622 raft_consensus.cc:729] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.927559 29627 raft_consensus.cc:348] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:44.927747 29627 raft_consensus.cc:374] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.927810 29627 raft_consensus.cc:729] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.927809 29622 consensus_queue.cc:260] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:44.928141 29627 consensus_queue.cc:260] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:44.928256 29622 ts_tablet_manager.cc:1398] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:44.928828 29627 ts_tablet_manager.cc:1398] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:44.929620 29623 ts_tablet_manager.cc:1398] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:17:44.929850 29623 tablet_bootstrap.cc:492] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:44.930511 29626 ts_tablet_manager.cc:1398] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:17:44.930856 29626 tablet_bootstrap.cc:492] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:44.931529 29623 tablet_bootstrap.cc:654] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.932551 29626 tablet_bootstrap.cc:654] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.935350 29623 tablet_bootstrap.cc:492] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:44.935508 29623 ts_tablet_manager.cc:1367] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240502 08:17:44.936432 29623 raft_consensus.cc:348] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:44.936661 29623 raft_consensus.cc:374] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.936743 29623 raft_consensus.cc:729] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.936952 29623 consensus_queue.cc:260] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:44.938120 29623 ts_tablet_manager.cc:1398] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:17:44.938405 29623 tablet_bootstrap.cc:492] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:44.940380 29623 tablet_bootstrap.cc:654] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.945866 29623 tablet_bootstrap.cc:492] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:44.946385 29623 ts_tablet_manager.cc:1367] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240502 08:17:44.947152 29623 raft_consensus.cc:348] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:44.947363 29623 raft_consensus.cc:374] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.947469 29623 raft_consensus.cc:729] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.947746 29626 tablet_bootstrap.cc:492] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:44.947687 29623 consensus_queue.cc:260] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:44.948078 29626 ts_tablet_manager.cc:1367] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.005s
I20240502 08:17:44.948523 29623 ts_tablet_manager.cc:1398] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:44.949019 29626 raft_consensus.cc:348] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:44.949258 29626 raft_consensus.cc:374] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.949338 29626 raft_consensus.cc:729] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.949579 29626 consensus_queue.cc:260] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:44.950232 29626 ts_tablet_manager.cc:1398] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:45.079015 29536 raft_consensus.cc:482] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:45.079181 29536 raft_consensus.cc:504] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:45.080013 29536 leader_election.cc:288] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:45.080381 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c312b65213444f9e946a932d98874c" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:45.080480 27256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c312b65213444f9e946a932d98874c" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:45.080616 27387 raft_consensus.cc:2454] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:45.080672 27256 raft_consensus.cc:2454] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:45.080986 27586 leader_election.cc:302] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:45.081216 29536 raft_consensus.cc:2781] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:45.081336 29536 raft_consensus.cc:482] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:45.081429 29536 raft_consensus.cc:3037] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:45.083307 29536 raft_consensus.cc:504] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:45.083810 29536 leader_election.cc:288] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:45.084205 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c312b65213444f9e946a932d98874c" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:45.084237 27256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c312b65213444f9e946a932d98874c" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:45.084404 27387 raft_consensus.cc:3037] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:45.084398 27256 raft_consensus.cc:3037] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:45.087126 27387 raft_consensus.cc:2454] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:45.087141 27256 raft_consensus.cc:2454] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:45.087472 27586 leader_election.cc:302] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:45.087734 29536 raft_consensus.cc:2781] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:45.088063 29536 raft_consensus.cc:686] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:17:45.088299 29536 consensus_queue.cc:237] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:45.091346 27113 catalog_manager.cc:5484] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:45.131237 29526 raft_consensus.cc:482] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:45.131369 29526 raft_consensus.cc:504] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:45.131863 29526 leader_election.cc:288] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:45.132234 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc33dff3dee4b3fbd0620b5d2faaea1" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:45.132459 27521 raft_consensus.cc:2454] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:45.132416 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc33dff3dee4b3fbd0620b5d2faaea1" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:45.132645 27387 raft_consensus.cc:2454] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:45.132856 27194 leader_election.cc:302] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:45.133096 29526 raft_consensus.cc:2781] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:45.133198 29526 raft_consensus.cc:482] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:45.133306 29526 raft_consensus.cc:3037] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:45.135393 29526 raft_consensus.cc:504] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:45.135879 29526 leader_election.cc:288] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:45.136193 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc33dff3dee4b3fbd0620b5d2faaea1" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:45.136399 27521 raft_consensus.cc:3037] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:45.136373 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc33dff3dee4b3fbd0620b5d2faaea1" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:45.136508 27387 raft_consensus.cc:3037] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:45.138213 27387 raft_consensus.cc:2454] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:45.138545 27193 leader_election.cc:302] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:45.138813 29526 raft_consensus.cc:2781] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:45.139014 29526 raft_consensus.cc:686] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:45.139008 27521 raft_consensus.cc:2454] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:45.139551 29526 consensus_queue.cc:237] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:45.143190 27113 catalog_manager.cc:5484] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:45.260025 29536 raft_consensus.cc:482] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:45.260232 29536 raft_consensus.cc:504] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:45.261032 29536 leader_election.cc:288] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:45.261348 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7d725414664b64b0ca58babe5df1ff" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:45.261560 27521 raft_consensus.cc:2454] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:45.261484 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7d725414664b64b0ca58babe5df1ff" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:45.261731 27388 raft_consensus.cc:2454] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:45.261907 27587 leader_election.cc:302] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:45.262246 29536 raft_consensus.cc:2781] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:45.262355 29536 raft_consensus.cc:482] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:45.262468 29536 raft_consensus.cc:3037] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:45.264318 29536 raft_consensus.cc:504] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:45.264780 29536 leader_election.cc:288] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:45.265177 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7d725414664b64b0ca58babe5df1ff" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:45.265192 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7d725414664b64b0ca58babe5df1ff" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:45.265347 27388 raft_consensus.cc:3037] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:45.265316 27521 raft_consensus.cc:3037] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:45.267148 27388 raft_consensus.cc:2454] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:45.267168 27521 raft_consensus.cc:2454] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:45.267496 27586 leader_election.cc:302] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:45.267743 29536 raft_consensus.cc:2781] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:45.267895 29536 raft_consensus.cc:686] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:17:45.268174 29536 consensus_queue.cc:237] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:45.271602 27113 catalog_manager.cc:5484] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:45.537616 29526 consensus_queue.cc:1035] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:45.544174 29612 consensus_queue.cc:1035] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:45.571164 29536 consensus_queue.cc:1035] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:45.578397 29650 consensus_queue.cc:1035] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:45.665419 29650 consensus_queue.cc:1035] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:45.667886 29640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:45.668455 29640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 08:17:45.669473 29536 consensus_queue.cc:1035] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:45.677232 29640 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
INSERT report
rows total: 2000
time total: 276.932 ms
time per row: 0.138466 ms
I20240502 08:17:46.094070 27116 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:46.102690 27236 tablet_service.cc:1467] Processing CreateTablet for tablet a31709aa6b2b4ff784114fb48cbc8f2a (DEFAULT_TABLE table=rebalance_test_table_17 [id=c179f8e244074d9ead49cbf91748c01e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:46.102687 27498 tablet_service.cc:1467] Processing CreateTablet for tablet a31709aa6b2b4ff784114fb48cbc8f2a (DEFAULT_TABLE table=rebalance_test_table_17 [id=c179f8e244074d9ead49cbf91748c01e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:46.103116 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a31709aa6b2b4ff784114fb48cbc8f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:46.103107 27236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a31709aa6b2b4ff784114fb48cbc8f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:46.103957 27630 tablet_service.cc:1467] Processing CreateTablet for tablet a31709aa6b2b4ff784114fb48cbc8f2a (DEFAULT_TABLE table=rebalance_test_table_17 [id=c179f8e244074d9ead49cbf91748c01e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:46.104331 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a31709aa6b2b4ff784114fb48cbc8f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:46.108278 27368 tablet_service.cc:1467] Processing CreateTablet for tablet a3724af3b82c43b0a1036f4d9703a3a9 (DEFAULT_TABLE table=rebalance_test_table_17 [id=c179f8e244074d9ead49cbf91748c01e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:46.108623 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3724af3b82c43b0a1036f4d9703a3a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:46.109573 29707 tablet_bootstrap.cc:492] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:46.109505 27237 tablet_service.cc:1467] Processing CreateTablet for tablet 3047a6db2c9d438eb5720d848b9ea5bd (DEFAULT_TABLE table=rebalance_test_table_17 [id=c179f8e244074d9ead49cbf91748c01e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:46.111249 29708 tablet_bootstrap.cc:492] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:46.111485 27499 tablet_service.cc:1467] Processing CreateTablet for tablet a3724af3b82c43b0a1036f4d9703a3a9 (DEFAULT_TABLE table=rebalance_test_table_17 [id=c179f8e244074d9ead49cbf91748c01e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:46.111853 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3724af3b82c43b0a1036f4d9703a3a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:46.111760 27498 tablet_service.cc:1467] Processing CreateTablet for tablet 3047a6db2c9d438eb5720d848b9ea5bd (DEFAULT_TABLE table=rebalance_test_table_17 [id=c179f8e244074d9ead49cbf91748c01e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:46.112298 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3047a6db2c9d438eb5720d848b9ea5bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:46.113787 29708 tablet_bootstrap.cc:654] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:46.114742 27629 tablet_service.cc:1467] Processing CreateTablet for tablet 3047a6db2c9d438eb5720d848b9ea5bd (DEFAULT_TABLE table=rebalance_test_table_17 [id=c179f8e244074d9ead49cbf91748c01e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:46.115142 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3047a6db2c9d438eb5720d848b9ea5bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:46.118100 27238 tablet_service.cc:1467] Processing CreateTablet for tablet a3724af3b82c43b0a1036f4d9703a3a9 (DEFAULT_TABLE table=rebalance_test_table_17 [id=c179f8e244074d9ead49cbf91748c01e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:46.118377 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3047a6db2c9d438eb5720d848b9ea5bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:46.121469 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3724af3b82c43b0a1036f4d9703a3a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:46.128504 29707 tablet_bootstrap.cc:654] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:46.141170 29707 tablet_bootstrap.cc:492] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:46.141319 29707 ts_tablet_manager.cc:1367] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.007s
I20240502 08:17:46.141399 29712 tablet_bootstrap.cc:492] T a3724af3b82c43b0a1036f4d9703a3a9 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:46.142313 29707 raft_consensus.cc:348] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:46.142545 29707 raft_consensus.cc:374] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:46.142621 29707 raft_consensus.cc:729] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:46.142895 29707 consensus_queue.cc:260] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:46.143569 29707 ts_tablet_manager.cc:1398] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:46.143579 29712 tablet_bootstrap.cc:654] T a3724af3b82c43b0a1036f4d9703a3a9 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:46.144047 29707 tablet_bootstrap.cc:492] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:46.145762 29707 tablet_bootstrap.cc:654] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:46.154088 29710 tablet_bootstrap.cc:492] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:46.154297 29708 tablet_bootstrap.cc:492] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:46.154453 29708 ts_tablet_manager.cc:1367] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.043s user 0.014s sys 0.019s
I20240502 08:17:46.154736 29707 tablet_bootstrap.cc:492] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:46.154886 29707 ts_tablet_manager.cc:1367] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20240502 08:17:46.155386 29708 raft_consensus.cc:348] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:46.155491 29707 raft_consensus.cc:348] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:46.155596 29708 raft_consensus.cc:374] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:46.155661 29707 raft_consensus.cc:374] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:46.155732 29707 raft_consensus.cc:729] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:46.155772 29710 tablet_bootstrap.cc:654] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:46.155853 29708 raft_consensus.cc:729] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:46.155962 29707 consensus_queue.cc:260] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:46.156289 29708 consensus_queue.cc:260] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:46.156499 29707 ts_tablet_manager.cc:1398] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:46.156761 29707 tablet_bootstrap.cc:492] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:46.156992 29708 ts_tablet_manager.cc:1398] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:46.157332 29708 tablet_bootstrap.cc:492] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:46.158259 29707 tablet_bootstrap.cc:654] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:46.158587 29712 tablet_bootstrap.cc:492] T a3724af3b82c43b0a1036f4d9703a3a9 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:46.158744 29712 ts_tablet_manager.cc:1367] T a3724af3b82c43b0a1036f4d9703a3a9 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240502 08:17:46.161166 29712 raft_consensus.cc:348] T a3724af3b82c43b0a1036f4d9703a3a9 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:46.161417 29712 raft_consensus.cc:374] T a3724af3b82c43b0a1036f4d9703a3a9 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:46.161514 29712 raft_consensus.cc:729] T a3724af3b82c43b0a1036f4d9703a3a9 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:46.161720 29712 consensus_queue.cc:260] T a3724af3b82c43b0a1036f4d9703a3a9 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:46.162295 29712 ts_tablet_manager.cc:1398] T a3724af3b82c43b0a1036f4d9703a3a9 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:17:46.163177 29708 tablet_bootstrap.cc:654] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:46.168658 29707 tablet_bootstrap.cc:492] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:46.168787 29707 ts_tablet_manager.cc:1367] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240502 08:17:46.169373 29707 raft_consensus.cc:348] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:46.169530 29707 raft_consensus.cc:374] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:46.169624 29707 raft_consensus.cc:729] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:46.169852 29707 consensus_queue.cc:260] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:46.170387 29707 ts_tablet_manager.cc:1398] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 08:17:46.179384 29708 tablet_bootstrap.cc:492] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:46.179399 29710 tablet_bootstrap.cc:492] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:46.179524 29710 ts_tablet_manager.cc:1367] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240502 08:17:46.179536 29708 ts_tablet_manager.cc:1367] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.014s
I20240502 08:17:46.180233 29708 raft_consensus.cc:348] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:46.180244 29710 raft_consensus.cc:348] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:46.180392 29708 raft_consensus.cc:374] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:46.180469 29708 raft_consensus.cc:729] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:46.180483 29710 raft_consensus.cc:374] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:46.180599 29710 raft_consensus.cc:729] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:46.180660 29708 consensus_queue.cc:260] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:46.180794 29710 consensus_queue.cc:260] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:46.181195 29708 ts_tablet_manager.cc:1398] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:46.181495 29708 tablet_bootstrap.cc:492] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:46.182556 29710 ts_tablet_manager.cc:1398] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:46.182850 29710 tablet_bootstrap.cc:492] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:46.183378 29708 tablet_bootstrap.cc:654] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:46.184346 29710 tablet_bootstrap.cc:654] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:46.202961 29710 tablet_bootstrap.cc:492] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:46.203127 29710 ts_tablet_manager.cc:1367] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.004s
I20240502 08:17:46.203256 29708 tablet_bootstrap.cc:492] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:46.203408 29708 ts_tablet_manager.cc:1367] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.016s
I20240502 08:17:46.203759 29710 raft_consensus.cc:348] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:46.203961 29710 raft_consensus.cc:374] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:46.204022 29710 raft_consensus.cc:729] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:46.203963 29708 raft_consensus.cc:348] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:46.204139 29708 raft_consensus.cc:374] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:46.204219 29708 raft_consensus.cc:729] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:46.204228 29710 consensus_queue.cc:260] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:46.204408 29708 consensus_queue.cc:260] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:46.204690 29710 ts_tablet_manager.cc:1398] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 08:17:46.204897 29708 ts_tablet_manager.cc:1398] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:46.224788 29637 raft_consensus.cc:482] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:46.224967 29637 raft_consensus.cc:504] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:46.225605 29637 leader_election.cc:288] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:46.226074 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3047a6db2c9d438eb5720d848b9ea5bd" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:46.226060 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3047a6db2c9d438eb5720d848b9ea5bd" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:46.226301 27651 raft_consensus.cc:2454] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:46.226346 27521 raft_consensus.cc:2454] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:46.226716 27194 leader_election.cc:302] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:46.227186 29637 raft_consensus.cc:2781] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:46.227299 29637 raft_consensus.cc:482] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:46.227397 29637 raft_consensus.cc:3037] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:46.230238 29637 raft_consensus.cc:504] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:46.230829 29637 leader_election.cc:288] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:46.231150 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3047a6db2c9d438eb5720d848b9ea5bd" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:46.231333 27651 raft_consensus.cc:3037] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:46.231287 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3047a6db2c9d438eb5720d848b9ea5bd" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:46.231509 27519 raft_consensus.cc:3037] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:46.233799 27519 raft_consensus.cc:2454] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:46.234099 27651 raft_consensus.cc:2454] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:46.234272 27194 leader_election.cc:302] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:46.234907 29637 raft_consensus.cc:2781] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:46.235073 29637 raft_consensus.cc:686] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:46.235358 29637 consensus_queue.cc:237] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:46.238746 27116 catalog_manager.cc:5484] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:46.310175 29635 raft_consensus.cc:482] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:46.310338 29635 raft_consensus.cc:504] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:46.310935 29635 leader_election.cc:288] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:46.311276 27256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31709aa6b2b4ff784114fb48cbc8f2a" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:46.311347 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31709aa6b2b4ff784114fb48cbc8f2a" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:46.311514 27256 raft_consensus.cc:2454] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:46.311589 27521 raft_consensus.cc:2454] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:46.311882 27586 leader_election.cc:302] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:46.312124 29635 raft_consensus.cc:2781] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:46.312228 29635 raft_consensus.cc:482] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:46.312295 29635 raft_consensus.cc:3037] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:46.314131 29635 raft_consensus.cc:504] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:46.314882 29635 leader_election.cc:288] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:46.314985 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31709aa6b2b4ff784114fb48cbc8f2a" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:46.315064 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31709aa6b2b4ff784114fb48cbc8f2a" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:46.315200 27258 raft_consensus.cc:3037] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:46.315297 27521 raft_consensus.cc:3037] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:46.316087 29637 raft_consensus.cc:482] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:46.316310 29637 raft_consensus.cc:504] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:46.317303 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3724af3b82c43b0a1036f4d9703a3a9" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:46.317412 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3724af3b82c43b0a1036f4d9703a3a9" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:46.317498 27258 raft_consensus.cc:2454] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:46.317521 27519 raft_consensus.cc:2454] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:46.317667 27390 raft_consensus.cc:2454] T a3724af3b82c43b0a1036f4d9703a3a9 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:46.317844 27586 leader_election.cc:302] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:46.317903 27194 leader_election.cc:302] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:46.318099 29637 leader_election.cc:288] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:46.318169 29635 raft_consensus.cc:2781] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:46.318253 29612 raft_consensus.cc:2781] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:46.318339 29635 raft_consensus.cc:686] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:17:46.317709 27521 raft_consensus.cc:2454] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:46.318388 29612 raft_consensus.cc:482] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:46.318468 29612 raft_consensus.cc:3037] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:46.318637 29635 consensus_queue.cc:237] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:46.320694 29612 raft_consensus.cc:504] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:46.321195 29612 leader_election.cc:288] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:46.321575 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3724af3b82c43b0a1036f4d9703a3a9" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:46.321583 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3724af3b82c43b0a1036f4d9703a3a9" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:46.321732 27521 raft_consensus.cc:3037] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:46.321748 27390 raft_consensus.cc:3037] T a3724af3b82c43b0a1036f4d9703a3a9 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:46.322280 27116 catalog_manager.cc:5484] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:46.324223 27521 raft_consensus.cc:2454] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:46.324271 27390 raft_consensus.cc:2454] T a3724af3b82c43b0a1036f4d9703a3a9 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:46.324626 27193 leader_election.cc:302] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:46.324925 29612 raft_consensus.cc:2781] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:46.325062 29612 raft_consensus.cc:686] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:46.325294 29612 consensus_queue.cc:237] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:46.328051 27116 catalog_manager.cc:5484] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:46.657037 29612 consensus_queue.cc:1035] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:46.663967 29526 consensus_queue.cc:1035] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:46.694342 29723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:46.694651 29723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:46.701958 29723 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240502 08:17:46.799820 29650 consensus_queue.cc:1035] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:46.825711 29650 consensus_queue.cc:1035] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:46.831490 29526 consensus_queue.cc:1035] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:46.850190 29637 consensus_queue.cc:1035] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 258.953 ms
time per row: 0.129477 ms
I20240502 08:17:47.132025 27110 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:47.141294 27368 tablet_service.cc:1467] Processing CreateTablet for tablet f733b0e3238f498cb54e3b2beef24249 (DEFAULT_TABLE table=rebalance_test_table_18 [id=9df8e288a677400882b593a841c5c051]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:47.141789 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f733b0e3238f498cb54e3b2beef24249. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:47.142038 27630 tablet_service.cc:1467] Processing CreateTablet for tablet f733b0e3238f498cb54e3b2beef24249 (DEFAULT_TABLE table=rebalance_test_table_18 [id=9df8e288a677400882b593a841c5c051]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:47.141979 27498 tablet_service.cc:1467] Processing CreateTablet for tablet a67cb9c98522464399a44186109333cf (DEFAULT_TABLE table=rebalance_test_table_18 [id=9df8e288a677400882b593a841c5c051]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:47.142434 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f733b0e3238f498cb54e3b2beef24249. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:47.142452 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67cb9c98522464399a44186109333cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:47.142431 27369 tablet_service.cc:1467] Processing CreateTablet for tablet a67cb9c98522464399a44186109333cf (DEFAULT_TABLE table=rebalance_test_table_18 [id=9df8e288a677400882b593a841c5c051]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:47.142771 27369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67cb9c98522464399a44186109333cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:47.146589 27629 tablet_service.cc:1467] Processing CreateTablet for tablet a67cb9c98522464399a44186109333cf (DEFAULT_TABLE table=rebalance_test_table_18 [id=9df8e288a677400882b593a841c5c051]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:47.146996 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67cb9c98522464399a44186109333cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:47.148908 27238 tablet_service.cc:1467] Processing CreateTablet for tablet f733b0e3238f498cb54e3b2beef24249 (DEFAULT_TABLE table=rebalance_test_table_18 [id=9df8e288a677400882b593a841c5c051]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:47.149323 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f733b0e3238f498cb54e3b2beef24249. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:47.154163 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 549102ad342342f49049f34ebac5bc7d (DEFAULT_TABLE table=rebalance_test_table_18 [id=9df8e288a677400882b593a841c5c051]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:47.154624 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549102ad342342f49049f34ebac5bc7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:47.158108 29789 tablet_bootstrap.cc:492] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:47.159551 27370 tablet_service.cc:1467] Processing CreateTablet for tablet 549102ad342342f49049f34ebac5bc7d (DEFAULT_TABLE table=rebalance_test_table_18 [id=9df8e288a677400882b593a841c5c051]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:47.159849 29789 tablet_bootstrap.cc:654] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:47.159901 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549102ad342342f49049f34ebac5bc7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:47.163166 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 549102ad342342f49049f34ebac5bc7d (DEFAULT_TABLE table=rebalance_test_table_18 [id=9df8e288a677400882b593a841c5c051]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:47.163545 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549102ad342342f49049f34ebac5bc7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:47.168316 29788 tablet_bootstrap.cc:492] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:47.171976 29788 tablet_bootstrap.cc:654] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:47.172652 29786 tablet_bootstrap.cc:492] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:47.174926 29787 tablet_bootstrap.cc:492] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:47.175154 29789 tablet_bootstrap.cc:492] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:47.175494 29789 ts_tablet_manager.cc:1367] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.006s
I20240502 08:17:47.176388 29789 raft_consensus.cc:348] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:47.176753 29789 raft_consensus.cc:374] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:47.176980 29789 raft_consensus.cc:729] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:47.177861 29789 consensus_queue.cc:260] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:47.178117 29788 tablet_bootstrap.cc:492] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:47.179087 29788 ts_tablet_manager.cc:1367] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240502 08:17:47.179656 29787 tablet_bootstrap.cc:654] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:47.179754 29789 ts_tablet_manager.cc:1398] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:17:47.181221 29787 tablet_bootstrap.cc:492] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:47.181345 29787 ts_tablet_manager.cc:1367] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 08:17:47.181907 29788 raft_consensus.cc:348] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:47.182199 29788 raft_consensus.cc:374] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:47.182116 29787 raft_consensus.cc:348] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:47.182334 29788 raft_consensus.cc:729] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:47.182412 29787 raft_consensus.cc:374] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:47.182549 29787 raft_consensus.cc:729] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:47.182652 29788 consensus_queue.cc:260] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:47.182868 29787 consensus_queue.cc:260] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:47.183569 29786 tablet_bootstrap.cc:654] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:47.184631 29789 tablet_bootstrap.cc:492] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:47.185170 29787 ts_tablet_manager.cc:1398] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:47.185437 29787 tablet_bootstrap.cc:492] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:47.186084 29789 tablet_bootstrap.cc:654] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:47.186939 29787 tablet_bootstrap.cc:654] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:47.189930 29786 tablet_bootstrap.cc:492] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:47.190128 29786 ts_tablet_manager.cc:1367] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240502 08:17:47.190899 29786 raft_consensus.cc:348] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:47.191143 29786 raft_consensus.cc:374] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:47.191216 29788 ts_tablet_manager.cc:1398] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240502 08:17:47.191277 29786 raft_consensus.cc:729] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:47.191542 29786 consensus_queue.cc:260] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:47.192076 29786 ts_tablet_manager.cc:1398] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:47.192338 29786 tablet_bootstrap.cc:492] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:47.193919 29786 tablet_bootstrap.cc:654] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:47.195586 29787 tablet_bootstrap.cc:492] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:47.195823 29787 ts_tablet_manager.cc:1367] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20240502 08:17:47.196302 29789 tablet_bootstrap.cc:492] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:47.196480 29789 ts_tablet_manager.cc:1367] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20240502 08:17:47.196714 29787 raft_consensus.cc:348] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:47.196898 29787 raft_consensus.cc:374] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:47.196969 29787 raft_consensus.cc:729] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:47.197232 29787 consensus_queue.cc:260] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:47.197268 29789 raft_consensus.cc:348] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:47.197463 29789 raft_consensus.cc:374] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:47.197531 29789 raft_consensus.cc:729] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:47.197752 29789 consensus_queue.cc:260] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:47.198963 29786 tablet_bootstrap.cc:492] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:47.198961 29787 ts_tablet_manager.cc:1398] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:17:47.199110 29786 ts_tablet_manager.cc:1367] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 08:17:47.199236 29787 tablet_bootstrap.cc:492] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:47.199811 29786 raft_consensus.cc:348] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:47.200016 29786 raft_consensus.cc:374] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:47.200088 29786 raft_consensus.cc:729] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:47.200323 29786 consensus_queue.cc:260] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:47.200657 29787 tablet_bootstrap.cc:654] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:47.200966 29786 ts_tablet_manager.cc:1398] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:47.202087 29786 tablet_bootstrap.cc:492] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:47.203637 29786 tablet_bootstrap.cc:654] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:47.202313 29789 ts_tablet_manager.cc:1398] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:17:47.210813 29787 tablet_bootstrap.cc:492] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:47.210951 29787 ts_tablet_manager.cc:1367] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240502 08:17:47.212678 29786 tablet_bootstrap.cc:492] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:47.212862 29786 ts_tablet_manager.cc:1367] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240502 08:17:47.211726 29787 raft_consensus.cc:348] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:47.213130 29787 raft_consensus.cc:374] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:47.213251 29787 raft_consensus.cc:729] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:47.213510 29787 consensus_queue.cc:260] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:47.213853 29786 raft_consensus.cc:348] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:47.214129 29786 raft_consensus.cc:374] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:47.214200 29787 ts_tablet_manager.cc:1398] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:17:47.214268 29786 raft_consensus.cc:729] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:47.214553 29786 consensus_queue.cc:260] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:47.215961 29786 ts_tablet_manager.cc:1398] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:47.256150 29776 raft_consensus.cc:482] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:47.256306 29776 raft_consensus.cc:504] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:47.256922 29776 leader_election.cc:288] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:47.257263 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549102ad342342f49049f34ebac5bc7d" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:47.257357 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549102ad342342f49049f34ebac5bc7d" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:47.257479 27652 raft_consensus.cc:2454] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:47.257573 27388 raft_consensus.cc:2454] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:47.257903 27194 leader_election.cc:302] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:47.258172 29776 raft_consensus.cc:2781] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:47.258280 29776 raft_consensus.cc:482] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:47.258369 29776 raft_consensus.cc:3037] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:47.260249 29776 raft_consensus.cc:504] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:47.260713 29776 leader_election.cc:288] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:47.261049 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549102ad342342f49049f34ebac5bc7d" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:47.261247 27652 raft_consensus.cc:3037] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:47.261205 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549102ad342342f49049f34ebac5bc7d" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:47.261400 27388 raft_consensus.cc:3037] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:47.264079 27652 raft_consensus.cc:2454] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:47.264189 27388 raft_consensus.cc:2454] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:47.264452 27194 leader_election.cc:302] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:47.264698 29776 raft_consensus.cc:2781] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:47.264832 29776 raft_consensus.cc:686] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:47.265108 29776 consensus_queue.cc:237] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:47.268621 27107 catalog_manager.cc:5484] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:47.327338 29645 raft_consensus.cc:482] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:47.327486 29645 raft_consensus.cc:504] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:47.328265 29645 leader_election.cc:288] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:47.328517 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67cb9c98522464399a44186109333cf" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:47.328666 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67cb9c98522464399a44186109333cf" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:47.328752 27652 raft_consensus.cc:2454] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:47.328891 27519 raft_consensus.cc:2454] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:47.329071 27326 leader_election.cc:302] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:47.329289 29645 raft_consensus.cc:2781] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:47.329406 29645 raft_consensus.cc:482] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:47.329489 29645 raft_consensus.cc:3037] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:47.331640 29645 raft_consensus.cc:504] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:47.332132 29645 leader_election.cc:288] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:47.332505 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67cb9c98522464399a44186109333cf" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:47.332597 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67cb9c98522464399a44186109333cf" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:47.332702 27652 raft_consensus.cc:3037] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:47.332793 27519 raft_consensus.cc:3037] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:47.335398 27519 raft_consensus.cc:2454] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:47.335451 27652 raft_consensus.cc:2454] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:47.335731 27326 leader_election.cc:302] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:47.335989 29645 raft_consensus.cc:2781] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:47.336133 29645 raft_consensus.cc:686] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:17:47.336395 29645 consensus_queue.cc:237] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:47.339802 27112 catalog_manager.cc:5484] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:47.460299 29776 raft_consensus.cc:482] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:47.460480 29776 raft_consensus.cc:504] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:47.460986 29776 leader_election.cc:288] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:47.461364 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f733b0e3238f498cb54e3b2beef24249" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:47.461601 27652 raft_consensus.cc:2454] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:47.461557 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f733b0e3238f498cb54e3b2beef24249" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:47.461755 27388 raft_consensus.cc:2454] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:47.462018 27194 leader_election.cc:302] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:47.462370 29776 raft_consensus.cc:2781] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:47.462478 29776 raft_consensus.cc:482] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:47.462581 29776 raft_consensus.cc:3037] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:47.464699 29776 raft_consensus.cc:504] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:47.465260 29776 leader_election.cc:288] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:47.465559 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f733b0e3238f498cb54e3b2beef24249" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:47.465749 27651 raft_consensus.cc:3037] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:47.465749 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f733b0e3238f498cb54e3b2beef24249" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:47.465950 27388 raft_consensus.cc:3037] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:47.467865 27651 raft_consensus.cc:2454] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:47.468202 27194 leader_election.cc:302] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:47.468436 29776 raft_consensus.cc:2781] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:47.468443 27388 raft_consensus.cc:2454] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:47.468580 29776 raft_consensus.cc:686] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:47.468859 29776 consensus_queue.cc:237] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:47.471874 27112 catalog_manager.cc:5484] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:47.645314 29776 consensus_queue.cc:1035] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:47.663394 29776 consensus_queue.cc:1035] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:47.764845 29645 consensus_queue.cc:1035] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:47.785507 29724 consensus_queue.cc:1035] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:47.888281 29802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:47.888619 29802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:47.896018 29802 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240502 08:17:48.004885 29805 consensus_queue.cc:1035] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:48.012694 29805 consensus_queue.cc:1035] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 254.717 ms
time per row: 0.127358 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:17:49.262517 27112 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:49.271414 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfa509d47e54b0f9ccacc9954981fb6 (DEFAULT_TABLE table=rebalance_test_table_19 [id=7f9af7f2cb49474eb721698045f06a63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.271863 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfa509d47e54b0f9ccacc9954981fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.271836 27498 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfa509d47e54b0f9ccacc9954981fb6 (DEFAULT_TABLE table=rebalance_test_table_19 [id=7f9af7f2cb49474eb721698045f06a63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.272248 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfa509d47e54b0f9ccacc9954981fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.272169 27369 tablet_service.cc:1467] Processing CreateTablet for tablet fee0386181d84b18803effc8af1bbbc4 (DEFAULT_TABLE table=rebalance_test_table_19 [id=7f9af7f2cb49474eb721698045f06a63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.272562 27369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee0386181d84b18803effc8af1bbbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.272528 27631 tablet_service.cc:1467] Processing CreateTablet for tablet fee0386181d84b18803effc8af1bbbc4 (DEFAULT_TABLE table=rebalance_test_table_19 [id=7f9af7f2cb49474eb721698045f06a63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.272922 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee0386181d84b18803effc8af1bbbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.277819 27499 tablet_service.cc:1467] Processing CreateTablet for tablet fee0386181d84b18803effc8af1bbbc4 (DEFAULT_TABLE table=rebalance_test_table_19 [id=7f9af7f2cb49474eb721698045f06a63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.278144 27368 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfa509d47e54b0f9ccacc9954981fb6 (DEFAULT_TABLE table=rebalance_test_table_19 [id=7f9af7f2cb49474eb721698045f06a63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.278244 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee0386181d84b18803effc8af1bbbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.278474 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfa509d47e54b0f9ccacc9954981fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.282596 27236 tablet_service.cc:1467] Processing CreateTablet for tablet 20a836dd442943d08a87d61afc785cd1 (DEFAULT_TABLE table=rebalance_test_table_19 [id=7f9af7f2cb49474eb721698045f06a63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.282999 27236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a836dd442943d08a87d61afc785cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.285145 29871 tablet_bootstrap.cc:492] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:49.285459 27629 tablet_service.cc:1467] Processing CreateTablet for tablet 20a836dd442943d08a87d61afc785cd1 (DEFAULT_TABLE table=rebalance_test_table_19 [id=7f9af7f2cb49474eb721698045f06a63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.285804 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a836dd442943d08a87d61afc785cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.286828 29871 tablet_bootstrap.cc:654] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.289260 29869 tablet_bootstrap.cc:492] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:49.291420 27500 tablet_service.cc:1467] Processing CreateTablet for tablet 20a836dd442943d08a87d61afc785cd1 (DEFAULT_TABLE table=rebalance_test_table_19 [id=7f9af7f2cb49474eb721698045f06a63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.291793 29869 tablet_bootstrap.cc:654] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.291905 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a836dd442943d08a87d61afc785cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.293952 29871 tablet_bootstrap.cc:492] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:49.294127 29871 ts_tablet_manager.cc:1367] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20240502 08:17:49.294644 29870 tablet_bootstrap.cc:492] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:49.295125 29871 raft_consensus.cc:348] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:49.295524 29871 raft_consensus.cc:374] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.295678 29871 raft_consensus.cc:729] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.296028 29871 consensus_queue.cc:260] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:49.296509 29870 tablet_bootstrap.cc:654] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.296706 29871 ts_tablet_manager.cc:1398] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:49.296998 29871 tablet_bootstrap.cc:492] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:49.298506 29871 tablet_bootstrap.cc:654] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.298938 29869 tablet_bootstrap.cc:492] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:49.299152 29869 ts_tablet_manager.cc:1367] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240502 08:17:49.300158 29869 raft_consensus.cc:348] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:49.300500 29869 raft_consensus.cc:374] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.300657 29869 raft_consensus.cc:729] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.300964 29869 consensus_queue.cc:260] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:49.301786 29869 ts_tablet_manager.cc:1398] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:49.302168 29869 tablet_bootstrap.cc:492] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:49.304435 29869 tablet_bootstrap.cc:654] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.304591 29870 tablet_bootstrap.cc:492] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:49.304826 29870 ts_tablet_manager.cc:1367] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240502 08:17:49.305696 29870 raft_consensus.cc:348] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:49.305943 29870 raft_consensus.cc:374] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.306090 29870 raft_consensus.cc:729] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.306361 29870 consensus_queue.cc:260] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:49.307003 29870 ts_tablet_manager.cc:1398] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:49.311472 29871 tablet_bootstrap.cc:492] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:49.311694 29871 ts_tablet_manager.cc:1367] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20240502 08:17:49.313083 29871 raft_consensus.cc:348] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:49.313482 29871 raft_consensus.cc:374] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.313627 29871 raft_consensus.cc:729] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.313979 29871 consensus_queue.cc:260] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:49.314699 29871 ts_tablet_manager.cc:1398] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:49.330933 29878 tablet_bootstrap.cc:492] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:49.332437 29878 tablet_bootstrap.cc:654] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.332412 29863 raft_consensus.cc:482] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:49.332559 29863 raft_consensus.cc:504] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:49.333271 29863 leader_election.cc:288] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:49.333833 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfa509d47e54b0f9ccacc9954981fb6" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:49.333988 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfa509d47e54b0f9ccacc9954981fb6" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:49.334223 29870 tablet_bootstrap.cc:492] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:49.334266 27388 raft_consensus.cc:2454] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:49.334264 27519 raft_consensus.cc:2454] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:49.334797 27194 leader_election.cc:302] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:49.335073 29863 raft_consensus.cc:2781] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:49.335193 29863 raft_consensus.cc:482] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:49.335310 29863 raft_consensus.cc:3037] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:49.337013 29870 tablet_bootstrap.cc:654] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.340130 29869 tablet_bootstrap.cc:492] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:49.340343 29869 ts_tablet_manager.cc:1367] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.024s
I20240502 08:17:49.341326 29869 raft_consensus.cc:348] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:49.341650 29869 raft_consensus.cc:374] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.341786 29869 raft_consensus.cc:729] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.341624 29863 raft_consensus.cc:504] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:49.342573 29863 leader_election.cc:288] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:49.342140 29869 consensus_queue.cc:260] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:49.343334 29869 ts_tablet_manager.cc:1398] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:17:49.343477 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfa509d47e54b0f9ccacc9954981fb6" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:49.343667 27519 raft_consensus.cc:3037] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:49.343700 29878 tablet_bootstrap.cc:492] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:49.343935 29878 ts_tablet_manager.cc:1367] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20240502 08:17:49.344679 29878 raft_consensus.cc:348] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:49.346163 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfa509d47e54b0f9ccacc9954981fb6" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:49.346383 27388 raft_consensus.cc:3037] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:49.347326 29878 raft_consensus.cc:374] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.347481 29878 raft_consensus.cc:729] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.347751 29878 consensus_queue.cc:260] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:49.348536 29878 ts_tablet_manager.cc:1398] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:17:49.348829 29878 tablet_bootstrap.cc:492] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:49.349049 27388 raft_consensus.cc:2454] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:49.351003 29878 tablet_bootstrap.cc:654] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.351157 27193 leader_election.cc:302] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:49.352815 29776 raft_consensus.cc:2781] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:49.352953 29776 raft_consensus.cc:686] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:49.353261 29776 consensus_queue.cc:237] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:49.356227 27519 raft_consensus.cc:2454] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:49.361948 29878 tablet_bootstrap.cc:492] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:49.362182 29878 ts_tablet_manager.cc:1367] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240502 08:17:49.363191 29878 raft_consensus.cc:348] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:49.363459 29878 raft_consensus.cc:374] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.363536 29878 raft_consensus.cc:729] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.363840 29878 consensus_queue.cc:260] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:49.364387 29878 ts_tablet_manager.cc:1398] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:49.365089 29870 tablet_bootstrap.cc:492] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:49.365247 29870 ts_tablet_manager.cc:1367] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.018s
I20240502 08:17:49.365996 29870 raft_consensus.cc:348] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:49.366266 29870 raft_consensus.cc:374] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.366338 29870 raft_consensus.cc:729] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.366573 29870 consensus_queue.cc:260] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:49.368225 29870 ts_tablet_manager.cc:1398] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:17:49.368526 29870 tablet_bootstrap.cc:492] T fee0386181d84b18803effc8af1bbbc4 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:49.370095 29870 tablet_bootstrap.cc:654] T fee0386181d84b18803effc8af1bbbc4 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.371783 27113 catalog_manager.cc:5484] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:49.374929 29870 tablet_bootstrap.cc:492] T fee0386181d84b18803effc8af1bbbc4 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:49.375051 29870 ts_tablet_manager.cc:1367] T fee0386181d84b18803effc8af1bbbc4 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240502 08:17:49.375624 29870 raft_consensus.cc:348] T fee0386181d84b18803effc8af1bbbc4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:49.375831 29870 raft_consensus.cc:374] T fee0386181d84b18803effc8af1bbbc4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.375926 29870 raft_consensus.cc:729] T fee0386181d84b18803effc8af1bbbc4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.376152 29870 consensus_queue.cc:260] T fee0386181d84b18803effc8af1bbbc4 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:49.376659 29870 ts_tablet_manager.cc:1398] T fee0386181d84b18803effc8af1bbbc4 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 08:17:49.424635 29864 raft_consensus.cc:482] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:49.424786 29864 raft_consensus.cc:504] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:49.425379 29864 leader_election.cc:288] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:49.425659 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee0386181d84b18803effc8af1bbbc4" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:49.425737 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee0386181d84b18803effc8af1bbbc4" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:49.425900 27519 raft_consensus.cc:2454] T fee0386181d84b18803effc8af1bbbc4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:49.425915 27652 raft_consensus.cc:2454] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:49.426347 27326 leader_election.cc:302] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:49.426623 29864 raft_consensus.cc:2781] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:49.426752 29864 raft_consensus.cc:482] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:49.426841 29864 raft_consensus.cc:3037] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:49.428738 29864 raft_consensus.cc:504] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:49.429251 29864 leader_election.cc:288] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:49.429544 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee0386181d84b18803effc8af1bbbc4" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:49.429630 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee0386181d84b18803effc8af1bbbc4" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:49.429744 27519 raft_consensus.cc:3037] T fee0386181d84b18803effc8af1bbbc4 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:49.429791 27652 raft_consensus.cc:3037] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:49.432421 27519 raft_consensus.cc:2454] T fee0386181d84b18803effc8af1bbbc4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:49.432453 27652 raft_consensus.cc:2454] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:49.432752 27326 leader_election.cc:302] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:49.432986 29864 raft_consensus.cc:2781] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:49.433146 29864 raft_consensus.cc:686] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:17:49.433390 29864 consensus_queue.cc:237] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:49.435736 29776 raft_consensus.cc:482] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:49.435910 29776 raft_consensus.cc:504] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:49.436560 29776 leader_election.cc:288] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:49.437005 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a836dd442943d08a87d61afc785cd1" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:49.437193 27519 raft_consensus.cc:2454] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:49.437203 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a836dd442943d08a87d61afc785cd1" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:49.437018 27113 catalog_manager.cc:5484] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:49.437520 27194 leader_election.cc:302] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:49.437772 27652 raft_consensus.cc:2454] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:49.437789 29776 raft_consensus.cc:2781] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:49.437881 29776 raft_consensus.cc:482] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:49.437988 29776 raft_consensus.cc:3037] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:49.440403 29776 raft_consensus.cc:504] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:49.440910 29776 leader_election.cc:288] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:49.441291 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a836dd442943d08a87d61afc785cd1" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:49.441349 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a836dd442943d08a87d61afc785cd1" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:49.441488 27519 raft_consensus.cc:3037] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:49.441511 27652 raft_consensus.cc:3037] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:49.443652 27652 raft_consensus.cc:2454] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:49.443655 27519 raft_consensus.cc:2454] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:49.444015 27194 leader_election.cc:302] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:49.444236 29776 raft_consensus.cc:2781] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:49.444391 29776 raft_consensus.cc:686] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:49.444669 29776 consensus_queue.cc:237] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:49.447489 27113 catalog_manager.cc:5484] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:49.782953 29886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:49.783282 29886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:49.791595 29886 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:17:49.819003 29863 consensus_queue.cc:1035] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_19'
I20240502 08:17:49.826994 29863 consensus_queue.cc:1035] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:49.837970 29645 consensus_queue.cc:1035] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:49.852964 29846 consensus_queue.cc:1035] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:49.853446 29908 consensus_queue.cc:1035] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:49.866377 29776 consensus_queue.cc:1035] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 197.116 ms
time per row: 0.0985582 ms
I20240502 08:17:50.174912 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.5:37815
--local_ip_for_outbound_sockets=127.20.207.5
--tserver_master_addrs=127.20.207.62:41499
--webserver_port=35485
--webserver_interface=127.20.207.5
--builtin_ntp_servers=127.20.207.20:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:17:50.506561 29945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:50.506839 29945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:50.506970 29945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:50.514608 29945 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:50.514720 29945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:50.514865 29945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.5
I20240502 08:17:50.524400 29945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.5:37815
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.207.5
--webserver_port=35485
--tserver_master_addrs=127.20.207.62:41499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.5
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:50.525629 29945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:50.527364 29945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:50.538897 29951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:50.539491 29954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:50.540236 29952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:50.540966 29945 server_base.cc:1034] running on GCE node
I20240502 08:17:50.541626 29945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:50.542742 29945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:50.543962 29945 hybrid_clock.cc:648] HybridClock initialized: now 1714637870543906 us; error 46 us; skew 500 ppm
I20240502 08:17:50.544263 29945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:50.546306 29945 webserver.cc:460] Webserver started at http://127.20.207.5:35485/ using document root <none> and password file <none>
I20240502 08:17:50.546968 29945 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:50.547087 29945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:50.552074 29945 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 08:17:50.554615 29960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:50.555788 29945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:17:50.555919 29945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1f0bf4bc3c5146539146b825b048bff3"
format_stamp: "Formatted at 2024-05-02 08:17:17 on dist-test-slave-169q"
I20240502 08:17:50.556677 29945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:50.580662 29945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:50.581437 29945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:50.581707 29945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:50.583127 29945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:50.584725 29945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:50.584812 29945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:50.584933 29945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:50.585000 29945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:50.620085 29945 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.5:37815
I20240502 08:17:50.620146 30072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.5:37815 every 8 connection(s)
I20240502 08:17:50.621606 29945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:17:50.624593 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 29945
I20240502 08:17:50.625141 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.6:38899
--local_ip_for_outbound_sockets=127.20.207.6
--tserver_master_addrs=127.20.207.62:41499
--webserver_port=37797
--webserver_interface=127.20.207.6
--builtin_ntp_servers=127.20.207.20:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:50.637825 30073 heartbeater.cc:347] Connected to a master server at 127.20.207.62:41499
I20240502 08:17:50.638195 30073 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:50.639205 30073 heartbeater.cc:511] Master 127.20.207.62:41499 requested a full tablet report, sending...
I20240502 08:17:50.641319 27110 ts_manager.cc:194] Re-registered known tserver with Master: 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:17:50.642318 27110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.5:36541
W20240502 08:17:50.997666 30077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:50.998073 30077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:50.998221 30077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:51.007746 30077 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:51.007879 30077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:51.008038 30077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.6
I20240502 08:17:51.018218 30077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.6:38899
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.207.6
--webserver_port=37797
--tserver_master_addrs=127.20.207.62:41499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.6
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:51.019474 30077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:51.021138 30077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:51.037288 30083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:51.038437 30084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:51.039352 30086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:51.040819 30077 server_base.cc:1034] running on GCE node
I20240502 08:17:51.041353 30077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:51.042369 30077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:51.043609 30077 hybrid_clock.cc:648] HybridClock initialized: now 1714637871043550 us; error 42 us; skew 500 ppm
I20240502 08:17:51.043902 30077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:51.046093 30077 webserver.cc:460] Webserver started at http://127.20.207.6:37797/ using document root <none> and password file <none>
I20240502 08:17:51.046875 30077 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:51.047017 30077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:51.052039 30077 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 08:17:51.054731 30092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:51.055943 30077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:17:51.056093 30077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "84cda20738d2452da166e31fc6da57a6"
format_stamp: "Formatted at 2024-05-02 08:17:17 on dist-test-slave-169q"
I20240502 08:17:51.056833 30077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:51.074829 30077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:51.075525 30077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:51.075753 30077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:51.076531 30077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:51.078086 30077 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:51.078156 30077 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:51.078263 30077 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:51.078312 30077 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:51.110342 30077 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.6:38899
I20240502 08:17:51.110432 30204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.6:38899 every 8 connection(s)
I20240502 08:17:51.111912 30077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:17:51.115962 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 30077
I20240502 08:17:51.116519 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.7:40343
--local_ip_for_outbound_sockets=127.20.207.7
--tserver_master_addrs=127.20.207.62:41499
--webserver_port=44619
--webserver_interface=127.20.207.7
--builtin_ntp_servers=127.20.207.20:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:51.124423 30205 heartbeater.cc:347] Connected to a master server at 127.20.207.62:41499
I20240502 08:17:51.124699 30205 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:51.125589 30205 heartbeater.cc:511] Master 127.20.207.62:41499 requested a full tablet report, sending...
I20240502 08:17:51.127373 27110 ts_manager.cc:194] Re-registered known tserver with Master: 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:51.128564 27110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.6:52781
W20240502 08:17:51.457135 30208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:51.457425 30208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:51.457566 30208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:51.466214 30208 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:51.466351 30208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:51.466491 30208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.7
I20240502 08:17:51.476271 30208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.7:40343
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.207.7
--webserver_port=44619
--tserver_master_addrs=127.20.207.62:41499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.7
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:51.477663 30208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:51.479426 30208 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:51.491163 30213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:51.491892 30214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:51.492762 30216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:51.493624 30208 server_base.cc:1034] running on GCE node
I20240502 08:17:51.494206 30208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:51.495038 30208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:51.496225 30208 hybrid_clock.cc:648] HybridClock initialized: now 1714637871496151 us; error 66 us; skew 500 ppm
I20240502 08:17:51.496543 30208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:51.498787 30208 webserver.cc:460] Webserver started at http://127.20.207.7:44619/ using document root <none> and password file <none>
I20240502 08:17:51.499533 30208 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:51.499655 30208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:51.504642 30208 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 08:17:51.507390 30222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:51.508591 30208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:17:51.508736 30208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
format_stamp: "Formatted at 2024-05-02 08:17:18 on dist-test-slave-169q"
I20240502 08:17:51.509572 30208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:51.539376 30208 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:51.540144 30208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:51.540405 30208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:51.541357 30208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:51.542805 30208 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:51.542892 30208 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:51.543006 30208 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:51.543063 30208 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:51.579077 30208 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.7:40343
I20240502 08:17:51.579138 30334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.7:40343 every 8 connection(s)
I20240502 08:17:51.581079 30208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 08:17:51.591317 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 30208
I20240502 08:17:51.592037 21308 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
/tmp/dist-test-taskZkl2nR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.207.8:41619
--local_ip_for_outbound_sockets=127.20.207.8
--tserver_master_addrs=127.20.207.62:41499
--webserver_port=45047
--webserver_interface=127.20.207.8
--builtin_ntp_servers=127.20.207.20:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:51.593088 30336 heartbeater.cc:347] Connected to a master server at 127.20.207.62:41499
I20240502 08:17:51.593354 30336 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:51.594167 30336 heartbeater.cc:511] Master 127.20.207.62:41499 requested a full tablet report, sending...
I20240502 08:17:51.595893 27110 ts_manager.cc:194] Re-registered known tserver with Master: 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:17:51.596870 27110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.7:53341
I20240502 08:17:51.646581 30073 heartbeater.cc:503] Master 127.20.207.62:41499 was elected leader, sending a full tablet report...
W20240502 08:17:51.934823 30340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:51.935213 30340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:51.935366 30340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:51.943215 30340 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:51.943357 30340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:51.943512 30340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.207.8
I20240502 08:17:51.952930 30340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.207.20:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.207.8:41619
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.207.8
--webserver_port=45047
--tserver_master_addrs=127.20.207.62:41499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.207.8
--log_dir=/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:51.954577 30340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:51.956804 30340 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:51.969295 30346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:51.970183 30345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:51.970788 30348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:51.971308 30340 server_base.cc:1034] running on GCE node
I20240502 08:17:51.971855 30340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:51.973184 30340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:51.974373 30340 hybrid_clock.cc:648] HybridClock initialized: now 1714637871974321 us; error 63 us; skew 500 ppm
I20240502 08:17:51.974673 30340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:51.986886 30340 webserver.cc:460] Webserver started at http://127.20.207.8:45047/ using document root <none> and password file <none>
I20240502 08:17:51.987605 30340 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:51.987725 30340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:51.993067 30340 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 08:17:51.996716 30354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:51.998226 30340 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:51.998376 30340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
format_stamp: "Formatted at 2024-05-02 08:17:18 on dist-test-slave-169q"
I20240502 08:17:51.999318 30340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:52.016636 30340 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:52.017370 30340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:52.017624 30340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:52.018741 30340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:52.020133 30340 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:52.020203 30340 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:52.020311 30340 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:52.020359 30340 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:52.056726 30340 rpc_server.cc:307] RPC server started. Bound to: 127.20.207.8:41619
I20240502 08:17:52.056759 30467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.207.8:41619 every 8 connection(s)
I20240502 08:17:52.058202 30340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 08:17:52.065223 21308 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkl2nR/build/asan/bin/kudu as pid 30340
I20240502 08:17:52.072260 30468 heartbeater.cc:347] Connected to a master server at 127.20.207.62:41499
I20240502 08:17:52.072549 30468 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:52.073451 30468 heartbeater.cc:511] Master 127.20.207.62:41499 requested a full tablet report, sending...
I20240502 08:17:52.074887 27110 ts_manager.cc:194] Re-registered known tserver with Master: 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:17:52.075616 27110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.207.8:45345
I20240502 08:17:52.077018 27110 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:17:52.087158 27369 tablet_service.cc:1467] Processing CreateTablet for tablet fae23f9cc6c446afae1e829dc5774b94 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0c5d03bf17d74da8bae4a472668ccb55]), partition=
I20240502 08:17:52.087529 27369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae23f9cc6c446afae1e829dc5774b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:52.116963 30008 tablet_service.cc:1467] Processing CreateTablet for tablet fae23f9cc6c446afae1e829dc5774b94 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0c5d03bf17d74da8bae4a472668ccb55]), partition=
I20240502 08:17:52.122032 30008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae23f9cc6c446afae1e829dc5774b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:52.131412 30205 heartbeater.cc:503] Master 127.20.207.62:41499 was elected leader, sending a full tablet report...
I20240502 08:17:52.143579 30484 tablet_bootstrap.cc:492] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:52.145776 30140 tablet_service.cc:1467] Processing CreateTablet for tablet fae23f9cc6c446afae1e829dc5774b94 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0c5d03bf17d74da8bae4a472668ccb55]), partition=
I20240502 08:17:52.146956 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae23f9cc6c446afae1e829dc5774b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:52.147843 30484 tablet_bootstrap.cc:654] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:52.155226 30485 tablet_bootstrap.cc:492] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:52.157743 30485 tablet_bootstrap.cc:654] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:52.160759 30485 log.cc:824] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:52.164227 30485 tablet_bootstrap.cc:492] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:52.164489 30485 ts_tablet_manager.cc:1367] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240502 08:17:52.166352 30484 tablet_bootstrap.cc:492] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:52.167385 30484 ts_tablet_manager.cc:1367] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.006s
I20240502 08:17:52.169011 30484 raft_consensus.cc:348] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:52.169433 30484 raft_consensus.cc:374] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:52.170557 30484 raft_consensus.cc:729] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:52.171451 30484 consensus_queue.cc:260] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:52.172573 30484 ts_tablet_manager.cc:1398] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:17:52.175984 30485 raft_consensus.cc:348] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:52.176379 30485 raft_consensus.cc:374] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:52.176699 30485 raft_consensus.cc:729] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:52.177348 30485 consensus_queue.cc:260] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:52.181447 30488 tablet_bootstrap.cc:492] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:52.186506 30485 ts_tablet_manager.cc:1398] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.022s user 0.007s sys 0.011s
I20240502 08:17:52.187424 30488 tablet_bootstrap.cc:654] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:52.189481 30488 log.cc:824] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:52.192483 30489 raft_consensus.cc:482] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:52.192785 30488 tablet_bootstrap.cc:492] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:17:52.192819 30489 raft_consensus.cc:504] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:52.193055 30488 ts_tablet_manager.cc:1367] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20240502 08:17:52.207615 30488 raft_consensus.cc:348] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:52.208020 30488 raft_consensus.cc:374] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:52.208106 30488 raft_consensus.cc:729] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:17:52.208825 30488 consensus_queue.cc:260] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:52.212302 30489 leader_election.cc:288] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:52.219241 30488 ts_tablet_manager.cc:1398] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.026s user 0.015s sys 0.002s
I20240502 08:17:52.226918 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae23f9cc6c446afae1e829dc5774b94" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:17:52.227365 30160 raft_consensus.cc:2454] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:17:52.228008 29964 leader_election.cc:302] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:52.228381 30489 raft_consensus.cc:2781] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:52.228893 30489 raft_consensus.cc:482] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:52.229168 30489 raft_consensus.cc:3037] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.231225 30078 raft_consensus.cc:482] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:52.231352 30078 raft_consensus.cc:504] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:52.232156 30078 leader_election.cc:288] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:52.232059 30489 raft_consensus.cc:504] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:52.233929 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae23f9cc6c446afae1e829dc5774b94" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:17:52.234144 30160 raft_consensus.cc:3037] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.235460 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae23f9cc6c446afae1e829dc5774b94" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:52.235678 27390 raft_consensus.cc:3037] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.235692 30489 leader_election.cc:288] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:52.236991 30160 raft_consensus.cc:2454] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:17:52.237370 29964 leader_election.cc:302] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:52.237715 30489 raft_consensus.cc:2781] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:52.238056 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae23f9cc6c446afae1e829dc5774b94" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:52.241091 27390 raft_consensus.cc:2454] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:17:52.247234 30489 raft_consensus.cc:686] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:17:52.247696 30489 consensus_queue.cc:237] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:52.260769 27110 catalog_manager.cc:5484] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:52.265717 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae23f9cc6c446afae1e829dc5774b94" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:17:52.266044 30160 raft_consensus.cc:2379] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83e171a5e16c4008923f1d43fc5ce7b1 in current term 1: Already voted for candidate 1f0bf4bc3c5146539146b825b048bff3 in this term.
I20240502 08:17:52.267277 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae23f9cc6c446afae1e829dc5774b94" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:17:52.267975 27323 leader_election.cc:302] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1; no voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6
I20240502 08:17:52.268268 30078 raft_consensus.cc:2726] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:17:52.364663 30206 tablet.cc:2369] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:17:52.374300 30074 tablet.cc:2369] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:52.375555 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:17:52.392521 27390 raft_consensus.cc:1261] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:52.392863 30160 raft_consensus.cc:1261] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:52.393597 30497 consensus_queue.cc:1035] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:52.394160 30489 consensus_queue.cc:1035] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:52.410701 27115 catalog_manager.cc:5576] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899) does not have the latest schema for tablet fae23f9cc6c446afae1e829dc5774b94 (table rebalancer_dynamic_table [id=0c5d03bf17d74da8bae4a472668ccb55]). Expected version 1 got 0
I20240502 08:17:52.411635 30501 mvcc.cc:205] Tried to move back new op lower bound from 7023156725312122880 to 7023156724730601472. Current Snapshot: MvccSnapshot[applied={T|T < 7023156725312122880}]
I20240502 08:17:52.414697 27110 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet fae23f9cc6c446afae1e829dc5774b94 (table rebalancer_dynamic_table [id=0c5d03bf17d74da8bae4a472668ccb55]). Expected version 1 got 0
I20240502 08:17:52.412989 30505 tablet.cc:1722] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:52.418252 30505 tablet.cc:1620] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:17:52.420547 30503 mvcc.cc:205] Tried to move back new op lower bound from 7023156725312122880 to 7023156724730601472. Current Snapshot: MvccSnapshot[applied={T|T < 7023156725312122880}]
I20240502 08:17:52.423671 30506 tablet.cc:1722] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:52.425230 27110 catalog_manager.cc:5576] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899) does not have the latest schema for tablet fae23f9cc6c446afae1e829dc5774b94 (table rebalancer_dynamic_table [id=0c5d03bf17d74da8bae4a472668ccb55]). Expected version 1 got 0
I20240502 08:17:52.426905 30506 tablet.cc:1620] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:17:52.439781 27115 catalog_manager.cc:5576] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291) does not have the latest schema for tablet fae23f9cc6c446afae1e829dc5774b94 (table rebalancer_dynamic_table [id=0c5d03bf17d74da8bae4a472668ccb55]). Expected version 1 got 0
I20240502 08:17:52.440752 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=0c5d03bf17d74da8bae4a472668ccb55] alter complete (version 1)
I20240502 08:17:52.444041 30508 tablet.cc:1722] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:52.445744 30508 tablet.cc:1620] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1: MemRowSet was empty: no flush needed.
I20240502 08:17:52.449200 27115 catalog_manager.cc:5576] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291) does not have the latest schema for tablet fae23f9cc6c446afae1e829dc5774b94 (table rebalancer_dynamic_table [id=0c5d03bf17d74da8bae4a472668ccb55]). Expected version 1 got 0
I20240502 08:17:52.552569 27115 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:17:52.560810 30505 tablet.cc:1722] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:52.561424 30505 tablet.cc:1620] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:17:52.561604 30506 tablet.cc:1722] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:52.561754 30508 tablet.cc:1722] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:52.562268 30506 tablet.cc:1620] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:17:52.562290 30508 tablet.cc:1620] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1: MemRowSet was empty: no flush needed.
I20240502 08:17:52.569300 27116 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:52.569653 27111 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 08:17:52.569862 27111 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 08:17:52.572012 27115 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=0c5d03bf17d74da8bae4a472668ccb55] alter complete (version 2)
I20240502 08:17:52.575060 27111 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 7f5a95b67d1c45578e323cc4ce8d4c14
I20240502 08:17:52.576161 27499 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f5a95b67d1c45578e323cc4ce8d4c14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:52 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:17:52.578274 30521 tablet_replica.cc:330] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:17:52.578565 27111 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet d465bff0ba8c42458e7308bb85adfc65
I20240502 08:17:52.579794 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f5a95b67d1c45578e323cc4ce8d4c14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:52 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:52.579226 27369 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f5a95b67d1c45578e323cc4ce8d4c14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:52 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:52.580202 27111 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet e15163867cbf4cc2ab2d6da3e430b261
I20240502 08:17:52.580673 27368 tablet_service.cc:1514] Processing DeleteTablet for tablet d465bff0ba8c42458e7308bb85adfc65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:52 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:52.580930 27370 tablet_service.cc:1514] Processing DeleteTablet for tablet e15163867cbf4cc2ab2d6da3e430b261 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:52 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:52.581801 27236 tablet_service.cc:1514] Processing DeleteTablet for tablet e15163867cbf4cc2ab2d6da3e430b261 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:52 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:52.581959 27629 tablet_service.cc:1514] Processing DeleteTablet for tablet e15163867cbf4cc2ab2d6da3e430b261 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:52 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:52.582199 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet d465bff0ba8c42458e7308bb85adfc65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:52 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:52.590845 30522 tablet_replica.cc:330] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:52.591459 30522 raft_consensus.cc:2229] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:52.591846 30522 raft_consensus.cc:2258] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:52.579479 30521 raft_consensus.cc:2229] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:52.581125 27499 tablet_service.cc:1514] Processing DeleteTablet for tablet d465bff0ba8c42458e7308bb85adfc65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:52 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:17:52.592718 30521 raft_consensus.cc:2258] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:52.594187 30521 ts_tablet_manager.cc:1875] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:52.595046 30008 tablet_service.cc:1467] Processing CreateTablet for tablet f5a0e83549644add82e39c5720805b95 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=3600173badac4b5bbac03472cd151621]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:52.595678 30008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a0e83549644add82e39c5720805b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:52.596040 27631 tablet_service.cc:1467] Processing CreateTablet for tablet e2b9488fe8ac46099b7e8adf4d0b9cd5 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=3600173badac4b5bbac03472cd151621]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:52.596041 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 81e73229c0d145e7bb81761bf4b820dc (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=3600173badac4b5bbac03472cd151621]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:52.596454 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81e73229c0d145e7bb81761bf4b820dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:52.596453 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2b9488fe8ac46099b7e8adf4d0b9cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:52.599604 30336 heartbeater.cc:503] Master 127.20.207.62:41499 was elected leader, sending a full tablet report...
I20240502 08:17:52.599483 30140 tablet_service.cc:1467] Processing CreateTablet for tablet 81e73229c0d145e7bb81761bf4b820dc (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=3600173badac4b5bbac03472cd151621]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:52.600030 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81e73229c0d145e7bb81761bf4b820dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:52.595503 27499 tablet_service.cc:1467] Processing CreateTablet for tablet e2b9488fe8ac46099b7e8adf4d0b9cd5 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=3600173badac4b5bbac03472cd151621]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:52.605660 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2b9488fe8ac46099b7e8adf4d0b9cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:52.617753 30488 tablet_bootstrap.cc:492] T 81e73229c0d145e7bb81761bf4b820dc P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:52.618445 30485 tablet_bootstrap.cc:492] T f5a0e83549644add82e39c5720805b95 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:52.618737 30529 tablet_bootstrap.cc:492] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:52.619560 30488 tablet_bootstrap.cc:654] T 81e73229c0d145e7bb81761bf4b820dc P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:52.620487 30485 tablet_bootstrap.cc:654] T f5a0e83549644add82e39c5720805b95 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:52.622225 30529 tablet_bootstrap.cc:654] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:52.628127 30523 tablet_replica.cc:330] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:52.628610 30523 raft_consensus.cc:2229] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:52.629040 30523 raft_consensus.cc:2258] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:52.629700 30528 tablet_bootstrap.cc:492] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:52.631793 30528 tablet_bootstrap.cc:654] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:52.636957 30522 ts_tablet_manager.cc:1875] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:52.636956 30523 ts_tablet_manager.cc:1875] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:52.646327 30485 tablet_bootstrap.cc:492] T f5a0e83549644add82e39c5720805b95 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:52.646520 30485 ts_tablet_manager.cc:1367] T f5a0e83549644add82e39c5720805b95 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.001s
I20240502 08:17:52.647714 30485 raft_consensus.cc:348] T f5a0e83549644add82e39c5720805b95 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:52.647934 30485 raft_consensus.cc:374] T f5a0e83549644add82e39c5720805b95 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:52.648021 30485 raft_consensus.cc:729] T f5a0e83549644add82e39c5720805b95 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:52.649068 30485 consensus_queue.cc:260] T f5a0e83549644add82e39c5720805b95 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:52.648811 30270 tablet_service.cc:1467] Processing CreateTablet for tablet 81e73229c0d145e7bb81761bf4b820dc (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=3600173badac4b5bbac03472cd151621]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:52.650297 30521 ts_tablet_manager.cc:1888] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:52.650457 30521 log.cc:1196] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/7f5a95b67d1c45578e323cc4ce8d4c14
I20240502 08:17:52.650498 30268 tablet_service.cc:1467] Processing CreateTablet for tablet e2b9488fe8ac46099b7e8adf4d0b9cd5 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=3600173badac4b5bbac03472cd151621]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:52.651052 30521 ts_tablet_manager.cc:1909] T 7f5a95b67d1c45578e323cc4ce8d4c14 P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:17:52.651376 30527 tablet_bootstrap.cc:492] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:52.652143 30269 tablet_service.cc:1467] Processing CreateTablet for tablet f5a0e83549644add82e39c5720805b95 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=3600173badac4b5bbac03472cd151621]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:52.652751 30528 tablet_bootstrap.cc:492] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:52.653190 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81e73229c0d145e7bb81761bf4b820dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:52.653826 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2b9488fe8ac46099b7e8adf4d0b9cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:52.654021 30269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a0e83549644add82e39c5720805b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:52.657699 30485 ts_tablet_manager.cc:1398] T f5a0e83549644add82e39c5720805b95 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240502 08:17:52.652901 30528 ts_tablet_manager.cc:1367] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240502 08:17:52.658355 30524 tablet_replica.cc:330] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:52.659905 30527 tablet_bootstrap.cc:654] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:52.658881 30528 raft_consensus.cc:348] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:52.660180 30528 raft_consensus.cc:374] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:52.660336 30528 raft_consensus.cc:729] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:52.660430 30524 raft_consensus.cc:2229] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:52.660612 30528 consensus_queue.cc:260] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:52.660887 30524 raft_consensus.cc:2258] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:52.662281 30528 ts_tablet_manager.cc:1398] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:52.659994 30529 tablet_bootstrap.cc:492] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:52.663020 30529 ts_tablet_manager.cc:1367] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20240502 08:17:52.663837 30529 raft_consensus.cc:348] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:52.664108 30529 raft_consensus.cc:374] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:52.664242 30529 raft_consensus.cc:729] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:52.664531 30529 consensus_queue.cc:260] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:52.665140 30529 ts_tablet_manager.cc:1398] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:52.666303 30524 ts_tablet_manager.cc:1875] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:52.668963 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 7f5a95b67d1c45578e323cc4ce8d4c14 (table rebalance_test_table_0 [id=a7bef71aae7647a990f6008892e44102]) successfully deleted
I20240502 08:17:52.675925 30521 tablet_replica.cc:330] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:17:52.676301 27116 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:17:52.676733 30521 raft_consensus.cc:2229] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:52.677563 30527 tablet_bootstrap.cc:492] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:52.677723 30527 ts_tablet_manager.cc:1367] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.004s
I20240502 08:17:52.678402 30527 raft_consensus.cc:348] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:52.678555 30527 raft_consensus.cc:374] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:52.678717 30527 raft_consensus.cc:729] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:52.678788 30523 ts_tablet_manager.cc:1888] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:52.678978 30523 log.cc:1196] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/7f5a95b67d1c45578e323cc4ce8d4c14
I20240502 08:17:52.679015 30527 consensus_queue.cc:260] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:52.679529 30523 ts_tablet_manager.cc:1909] T 7f5a95b67d1c45578e323cc4ce8d4c14 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:52.682364 30521 raft_consensus.cc:2258] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:52.683100 30521 ts_tablet_manager.cc:1875] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:52.684095 30527 ts_tablet_manager.cc:1398] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:17:52.686295 30505 tablet.cc:1722] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:52.688982 30523 tablet_replica.cc:330] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:52.689189 30523 raft_consensus.cc:2229] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:52.689402 30523 raft_consensus.cc:2258] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:52.690279 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 7f5a95b67d1c45578e323cc4ce8d4c14 (table rebalance_test_table_0 [id=a7bef71aae7647a990f6008892e44102]) successfully deleted
I20240502 08:17:52.690678 29863 raft_consensus.cc:482] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:52.690805 30488 tablet_bootstrap.cc:492] T 81e73229c0d145e7bb81761bf4b820dc P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:17:52.690821 29863 raft_consensus.cc:504] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:52.690950 30488 ts_tablet_manager.cc:1367] T 81e73229c0d145e7bb81761bf4b820dc P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.073s user 0.009s sys 0.000s
I20240502 08:17:52.691689 29863 leader_election.cc:288] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:52.691942 30488 raft_consensus.cc:348] T 81e73229c0d145e7bb81761bf4b820dc P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:52.692692 30488 raft_consensus.cc:374] T 81e73229c0d145e7bb81761bf4b820dc P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:52.692786 30488 raft_consensus.cc:729] T 81e73229c0d145e7bb81761bf4b820dc P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:17:52.693090 30488 consensus_queue.cc:260] T 81e73229c0d145e7bb81761bf4b820dc P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:52.693250 30521 ts_tablet_manager.cc:1888] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:52.693382 30521 log.cc:1196] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/d465bff0ba8c42458e7308bb85adfc65
I20240502 08:17:52.693778 30521 ts_tablet_manager.cc:1909] T d465bff0ba8c42458e7308bb85adfc65 P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:17:52.694057 30523 ts_tablet_manager.cc:1875] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:52.694370 30488 ts_tablet_manager.cc:1398] T 81e73229c0d145e7bb81761bf4b820dc P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:17:52.695137 30524 ts_tablet_manager.cc:1888] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:52.695350 30524 log.cc:1196] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/e15163867cbf4cc2ab2d6da3e430b261
I20240502 08:17:52.695516 30508 tablet.cc:1722] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:52.696244 30524 ts_tablet_manager.cc:1909] T e15163867cbf4cc2ab2d6da3e430b261 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:52.696530 30506 tablet.cc:1722] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:52.700042 30402 tablet_service.cc:1467] Processing CreateTablet for tablet f5a0e83549644add82e39c5720805b95 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=3600173badac4b5bbac03472cd151621]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:52.701432 30524 tablet_replica.cc:330] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:52.701591 30402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a0e83549644add82e39c5720805b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:52.702185 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet e15163867cbf4cc2ab2d6da3e430b261 (table rebalance_test_table_0 [id=a7bef71aae7647a990f6008892e44102]) successfully deleted
I20240502 08:17:52.702277 30524 raft_consensus.cc:2229] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:52.702630 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet d465bff0ba8c42458e7308bb85adfc65 (table rebalance_test_table_0 [id=a7bef71aae7647a990f6008892e44102]) successfully deleted
I20240502 08:17:52.702884 30524 raft_consensus.cc:2258] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:52.709587 30522 ts_tablet_manager.cc:1888] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:52.709745 30522 log.cc:1196] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/7f5a95b67d1c45578e323cc4ce8d4c14
I20240502 08:17:52.711177 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=0c5d03bf17d74da8bae4a472668ccb55] alter complete (version 3)
I20240502 08:17:52.713853 30535 tablet_bootstrap.cc:492] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:52.713943 30524 ts_tablet_manager.cc:1875] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:52.714769 30522 ts_tablet_manager.cc:1909] T 7f5a95b67d1c45578e323cc4ce8d4c14 P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:52.716516 30523 ts_tablet_manager.cc:1888] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:52.717676 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 7f5a95b67d1c45578e323cc4ce8d4c14 (table rebalance_test_table_0 [id=a7bef71aae7647a990f6008892e44102]) successfully deleted
I20240502 08:17:52.720724 30522 tablet_replica.cc:330] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:52.721016 30522 raft_consensus.cc:2229] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:52.721305 30522 raft_consensus.cc:2258] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:52.721454 30535 tablet_bootstrap.cc:654] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:52.722265 30522 ts_tablet_manager.cc:1875] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:52.722513 30523 log.cc:1196] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/d465bff0ba8c42458e7308bb85adfc65
I20240502 08:17:52.723070 30523 ts_tablet_manager.cc:1909] T d465bff0ba8c42458e7308bb85adfc65 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:52.724016 30523 tablet_replica.cc:330] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:52.724244 30523 raft_consensus.cc:2229] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:52.724731 30523 raft_consensus.cc:2258] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:52.724910 30535 log.cc:824] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:52.725464 30523 ts_tablet_manager.cc:1875] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:52.725819 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet d465bff0ba8c42458e7308bb85adfc65 (table rebalance_test_table_0 [id=a7bef71aae7647a990f6008892e44102]) successfully deleted
I20240502 08:17:52.733206 30523 ts_tablet_manager.cc:1888] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:52.733347 30523 log.cc:1196] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/e15163867cbf4cc2ab2d6da3e430b261
I20240502 08:17:52.733990 30523 ts_tablet_manager.cc:1909] T e15163867cbf4cc2ab2d6da3e430b261 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:52.735160 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet e15163867cbf4cc2ab2d6da3e430b261 (table rebalance_test_table_0 [id=a7bef71aae7647a990f6008892e44102]) successfully deleted
I20240502 08:17:52.748107 30524 ts_tablet_manager.cc:1888] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:52.749899 30522 ts_tablet_manager.cc:1888] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:52.751273 30524 log.cc:1196] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/d465bff0ba8c42458e7308bb85adfc65
I20240502 08:17:52.753159 30522 log.cc:1196] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/e15163867cbf4cc2ab2d6da3e430b261
I20240502 08:17:52.754287 30524 ts_tablet_manager.cc:1909] T d465bff0ba8c42458e7308bb85adfc65 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:52.755369 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet d465bff0ba8c42458e7308bb85adfc65 (table rebalance_test_table_0 [id=a7bef71aae7647a990f6008892e44102]) successfully deleted
I20240502 08:17:52.756982 30499 raft_consensus.cc:482] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:52.757134 30499 raft_consensus.cc:504] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:52.757895 30499 leader_election.cc:288] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:52.758831 30539 tablet_bootstrap.cc:492] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:17:52.759677 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b9488fe8ac46099b7e8adf4d0b9cd5" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:52.759902 27651 raft_consensus.cc:2454] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:17:52.760535 27456 leader_election.cc:302] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:52.761466 30522 ts_tablet_manager.cc:1909] T e15163867cbf4cc2ab2d6da3e430b261 P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:52.761557 30499 raft_consensus.cc:2781] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:52.761687 30499 raft_consensus.cc:482] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:52.761770 30499 raft_consensus.cc:3037] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.762918 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet e15163867cbf4cc2ab2d6da3e430b261 (table rebalance_test_table_0 [id=a7bef71aae7647a990f6008892e44102]) successfully deleted
I20240502 08:17:52.766112 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e73229c0d145e7bb81761bf4b820dc" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
W20240502 08:17:52.770176 27191 leader_election.cc:341] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:52.772058 30499 raft_consensus.cc:504] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:52.773238 30499 leader_election.cc:288] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:52.773258 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b9488fe8ac46099b7e8adf4d0b9cd5" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:52.773455 27651 raft_consensus.cc:3037] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.773499 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e73229c0d145e7bb81761bf4b820dc" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:17:52.773702 30159 raft_consensus.cc:2454] T 81e73229c0d145e7bb81761bf4b820dc P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:52.775633 27651 raft_consensus.cc:2454] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:52.775974 30535 tablet_bootstrap.cc:492] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:52.776065 27456 leader_election.cc:302] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:52.776353 30499 raft_consensus.cc:2781] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:52.776352 30535 ts_tablet_manager.cc:1367] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.063s user 0.019s sys 0.013s
I20240502 08:17:52.776486 30499 raft_consensus.cc:686] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:17:52.776546 30539 tablet_bootstrap.cc:654] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:52.776768 30499 consensus_queue.cc:237] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:52.777855 30539 log.cc:824] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:52.778934 27194 leader_election.cc:302] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 84cda20738d2452da166e31fc6da57a6; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:17:52.779538 29863 raft_consensus.cc:2781] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:52.779695 29863 raft_consensus.cc:482] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:52.779814 29863 raft_consensus.cc:3037] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.783447 30535 raft_consensus.cc:348] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:52.783849 30535 raft_consensus.cc:374] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:52.784009 30535 raft_consensus.cc:729] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:52.784816 30535 consensus_queue.cc:260] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:52.785517 29863 raft_consensus.cc:504] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:52.787869 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e73229c0d145e7bb81761bf4b820dc" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:17:52.788288 29863 leader_election.cc:288] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:52.788336 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e73229c0d145e7bb81761bf4b820dc" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:17:52.788559 30159 raft_consensus.cc:3037] T 81e73229c0d145e7bb81761bf4b820dc P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.789217 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b9488fe8ac46099b7e8adf4d0b9cd5" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:17:52.789572 30289 raft_consensus.cc:3037] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.790272 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b9488fe8ac46099b7e8adf4d0b9cd5" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:17:52.792248 30159 raft_consensus.cc:2454] T 81e73229c0d145e7bb81761bf4b820dc P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:52.792693 27194 leader_election.cc:302] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:52.793547 30500 raft_consensus.cc:2781] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:52.793725 30500 raft_consensus.cc:686] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:52.794106 30500 consensus_queue.cc:237] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:52.795020 30289 raft_consensus.cc:2454] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:17:52.806547 30539 tablet_bootstrap.cc:492] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:17:52.807099 30539 ts_tablet_manager.cc:1367] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.003s
I20240502 08:17:52.818290 30535 ts_tablet_manager.cc:1398] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.042s user 0.021s sys 0.015s
I20240502 08:17:52.818768 30535 tablet_bootstrap.cc:492] T f5a0e83549644add82e39c5720805b95 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:52.821918 27116 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:17:52.822145 27116 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:17:52.828008 27116 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet fae23f9cc6c446afae1e829dc5774b94
I20240502 08:17:52.828964 30008 tablet_service.cc:1514] Processing DeleteTablet for tablet fae23f9cc6c446afae1e829dc5774b94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:52 UTC) from {username='slave'} at 127.0.0.1:58374
I20240502 08:17:52.829804 27369 tablet_service.cc:1514] Processing DeleteTablet for tablet fae23f9cc6c446afae1e829dc5774b94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:52 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:52.830088 30523 tablet_replica.cc:330] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:52.829918 30140 tablet_service.cc:1514] Processing DeleteTablet for tablet fae23f9cc6c446afae1e829dc5774b94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:52 UTC) from {username='slave'} at 127.0.0.1:48808
I20240502 08:17:52.830288 30523 raft_consensus.cc:2229] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:52.830495 30523 raft_consensus.cc:2258] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:52.832082 30523 ts_tablet_manager.cc:1875] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:17:52.833599 30337 tablet.cc:2369] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:52.840276 30523 ts_tablet_manager.cc:1888] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:52.840495 30523 log.cc:1196] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/fae23f9cc6c446afae1e829dc5774b94
I20240502 08:17:52.840963 30523 ts_tablet_manager.cc:1909] T fae23f9cc6c446afae1e829dc5774b94 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:52.842478 30535 tablet_bootstrap.cc:654] T f5a0e83549644add82e39c5720805b95 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:52.843063 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet fae23f9cc6c446afae1e829dc5774b94 (table rebalancer_dynamic_table_new_name [id=0c5d03bf17d74da8bae4a472668ccb55]) successfully deleted
I20240502 08:17:52.845021 27116 catalog_manager.cc:5484] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:52.845129 27191 leader_election.cc:341] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:52.848979 30539 raft_consensus.cc:348] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:52.849462 30539 raft_consensus.cc:374] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:52.849573 30539 raft_consensus.cc:729] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:17:52.850395 30551 tablet_replica.cc:330] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:17:52.850855 30551 raft_consensus.cc:2229] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:52.851176 30539 consensus_queue.cc:260] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:52.851514 30551 raft_consensus.cc:2258] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:52.852983 30551 ts_tablet_manager.cc:1875] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:52.853436 30550 tablet_replica.cc:330] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:17:52.853864 30550 raft_consensus.cc:2229] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:52.855783 27116 catalog_manager.cc:5484] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:52.860291 30550 raft_consensus.cc:2258] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:52.862433 30535 tablet_bootstrap.cc:492] T f5a0e83549644add82e39c5720805b95 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:52.862609 30535 ts_tablet_manager.cc:1367] T f5a0e83549644add82e39c5720805b95 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.002s
I20240502 08:17:52.863515 30535 raft_consensus.cc:348] T f5a0e83549644add82e39c5720805b95 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:52.863754 30535 raft_consensus.cc:374] T f5a0e83549644add82e39c5720805b95 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:52.863883 30535 raft_consensus.cc:729] T f5a0e83549644add82e39c5720805b95 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:52.864260 30535 consensus_queue.cc:260] T f5a0e83549644add82e39c5720805b95 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:52.867775 30535 ts_tablet_manager.cc:1398] T f5a0e83549644add82e39c5720805b95 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:17:52.870944 30535 tablet_bootstrap.cc:492] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:52.877151 30550 ts_tablet_manager.cc:1875] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:52.879458 30535 tablet_bootstrap.cc:654] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:52.885001 30468 heartbeater.cc:503] Master 127.20.207.62:41499 was elected leader, sending a full tablet report...
I20240502 08:17:52.885506 30554 raft_consensus.cc:482] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:52.886711 30554 raft_consensus.cc:504] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:52.893376 30539 ts_tablet_manager.cc:1398] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.086s user 0.017s sys 0.007s
I20240502 08:17:52.893626 30551 ts_tablet_manager.cc:1888] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:52.893786 30551 log.cc:1196] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/fae23f9cc6c446afae1e829dc5774b94
I20240502 08:17:52.894516 30551 ts_tablet_manager.cc:1909] T fae23f9cc6c446afae1e829dc5774b94 P 84cda20738d2452da166e31fc6da57a6: Deleting consensus metadata
I20240502 08:17:52.896289 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet fae23f9cc6c446afae1e829dc5774b94 (table rebalancer_dynamic_table_new_name [id=0c5d03bf17d74da8bae4a472668ccb55]) successfully deleted
I20240502 08:17:52.899060 30535 tablet_bootstrap.cc:492] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:52.899267 30535 ts_tablet_manager.cc:1367] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.003s
I20240502 08:17:52.900509 30554 leader_election.cc:288] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:17:52.900174 30535 raft_consensus.cc:348] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:52.900799 30535 raft_consensus.cc:374] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:52.900882 30535 raft_consensus.cc:729] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:52.901134 30535 consensus_queue.cc:260] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:52.903728 30535 ts_tablet_manager.cc:1398] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240502 08:17:52.905200 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a0e83549644add82e39c5720805b95" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:17:52.905423 30289 raft_consensus.cc:2454] T f5a0e83549644add82e39c5720805b95 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:17:52.906409 30355 leader_election.cc:302] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:17:52.906571 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a0e83549644add82e39c5720805b95" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:17:52.906857 30028 raft_consensus.cc:2454] T f5a0e83549644add82e39c5720805b95 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:17:52.908675 30550 ts_tablet_manager.cc:1888] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:52.908811 30550 log.cc:1196] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/fae23f9cc6c446afae1e829dc5774b94
I20240502 08:17:52.909345 30550 ts_tablet_manager.cc:1909] T fae23f9cc6c446afae1e829dc5774b94 P 1f0bf4bc3c5146539146b825b048bff3: Deleting consensus metadata
I20240502 08:17:52.910997 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet fae23f9cc6c446afae1e829dc5774b94 (table rebalancer_dynamic_table_new_name [id=0c5d03bf17d74da8bae4a472668ccb55]) successfully deleted
I20240502 08:17:52.911190 30554 raft_consensus.cc:2781] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:52.911317 30554 raft_consensus.cc:482] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:52.911382 30554 raft_consensus.cc:3037] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.913573 30554 raft_consensus.cc:504] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:52.914516 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a0e83549644add82e39c5720805b95" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:17:52.914726 30289 raft_consensus.cc:3037] T f5a0e83549644add82e39c5720805b95 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.914696 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a0e83549644add82e39c5720805b95" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:17:52.914882 30028 raft_consensus.cc:3037] T f5a0e83549644add82e39c5720805b95 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.917385 30289 raft_consensus.cc:2454] T f5a0e83549644add82e39c5720805b95 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:17:52.917578 30028 raft_consensus.cc:2454] T f5a0e83549644add82e39c5720805b95 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:17:52.917791 30355 leader_election.cc:302] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:17:52.918085 30554 leader_election.cc:288] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:17:52.918262 30557 raft_consensus.cc:2781] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:52.918421 30557 raft_consensus.cc:686] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:17:52.918987 30557 consensus_queue.cc:237] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:52.923838 27116 catalog_manager.cc:5484] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:52.932597 27116 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:17:52.940740 30402 tablet_service.cc:1467] Processing CreateTablet for tablet b9d5687f84714fc7a5f3885c8efb272b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f86da6ea782d4a358678424e91733096]), partition=
I20240502 08:17:52.941107 30402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d5687f84714fc7a5f3885c8efb272b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:52.945912 30008 tablet_service.cc:1467] Processing CreateTablet for tablet b9d5687f84714fc7a5f3885c8efb272b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f86da6ea782d4a358678424e91733096]), partition=
I20240502 08:17:52.946478 30008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d5687f84714fc7a5f3885c8efb272b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:52.950767 30140 tablet_service.cc:1467] Processing CreateTablet for tablet b9d5687f84714fc7a5f3885c8efb272b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f86da6ea782d4a358678424e91733096]), partition=
I20240502 08:17:52.951107 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d5687f84714fc7a5f3885c8efb272b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:52.956125 30539 tablet_bootstrap.cc:492] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:17:52.957720 30539 tablet_bootstrap.cc:654] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:52.959650 30488 tablet_bootstrap.cc:492] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:52.960584 30485 tablet_bootstrap.cc:492] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:52.961658 30488 tablet_bootstrap.cc:654] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:52.962283 30485 tablet_bootstrap.cc:654] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:52.971798 30485 tablet_bootstrap.cc:492] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:52.971985 30485 ts_tablet_manager.cc:1367] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20240502 08:17:52.972548 30485 raft_consensus.cc:348] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:52.972767 30485 raft_consensus.cc:374] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:52.972843 30485 raft_consensus.cc:729] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:52.973094 30485 consensus_queue.cc:260] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:52.973769 30485 ts_tablet_manager.cc:1398] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:52.976197 30488 tablet_bootstrap.cc:492] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:17:52.976423 30488 ts_tablet_manager.cc:1367] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.008s
I20240502 08:17:52.979588 30488 raft_consensus.cc:348] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:52.979897 30488 raft_consensus.cc:374] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:52.979976 30488 raft_consensus.cc:729] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:17:52.980355 30488 consensus_queue.cc:260] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:52.981051 30488 ts_tablet_manager.cc:1398] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240502 08:17:52.986534 30539 tablet_bootstrap.cc:492] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:17:52.986701 30539 ts_tablet_manager.cc:1367] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.021s
I20240502 08:17:52.987730 30539 raft_consensus.cc:348] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:52.987907 30539 raft_consensus.cc:374] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:52.987993 30539 raft_consensus.cc:729] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:17:52.988224 30539 consensus_queue.cc:260] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:52.988595 30539 ts_tablet_manager.cc:1398] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240502 08:17:53.061448 30469 tablet.cc:2369] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:53.083839 27116 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 08:17:53.084319 27116 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 08:17:53.092262 27116 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 9babdea7835440909cf3763ca9ffcb64
I20240502 08:17:53.093195 27369 tablet_service.cc:1514] Processing DeleteTablet for tablet 9babdea7835440909cf3763ca9ffcb64 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:53.093514 30523 tablet_replica.cc:330] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:53.093730 30523 raft_consensus.cc:2229] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:53.094059 30523 raft_consensus.cc:2258] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.094135 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet 9babdea7835440909cf3763ca9ffcb64 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:53.094408 30522 tablet_replica.cc:330] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:53.094594 30522 raft_consensus.cc:2229] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:53.094835 30522 raft_consensus.cc:2258] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.094854 30523 ts_tablet_manager.cc:1875] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:53.095472 30522 ts_tablet_manager.cc:1875] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:53.102280 30523 ts_tablet_manager.cc:1888] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:53.102473 30523 log.cc:1196] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/9babdea7835440909cf3763ca9ffcb64
I20240502 08:17:53.102483 27236 tablet_service.cc:1514] Processing DeleteTablet for tablet 9babdea7835440909cf3763ca9ffcb64 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:53.102661 30522 ts_tablet_manager.cc:1888] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:53.102788 30522 log.cc:1196] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/9babdea7835440909cf3763ca9ffcb64
I20240502 08:17:53.102840 30524 tablet_replica.cc:330] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:53.103019 27116 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 82d6bd3211e34b2e85231b09b95e1587
I20240502 08:17:53.103160 30524 raft_consensus.cc:2229] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:53.103317 30523 ts_tablet_manager.cc:1909] T 9babdea7835440909cf3763ca9ffcb64 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:53.103545 30524 raft_consensus.cc:2258] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.103677 27499 tablet_service.cc:1514] Processing DeleteTablet for tablet 82d6bd3211e34b2e85231b09b95e1587 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:17:53.103956 30521 tablet_replica.cc:330] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:17:53.104120 27369 tablet_service.cc:1514] Processing DeleteTablet for tablet 82d6bd3211e34b2e85231b09b95e1587 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:53.104161 30521 raft_consensus.cc:2229] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:53.104318 30524 ts_tablet_manager.cc:1875] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:53.104425 30521 raft_consensus.cc:2258] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.105065 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet 82d6bd3211e34b2e85231b09b95e1587 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:53.105113 30521 ts_tablet_manager.cc:1875] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:53.105387 30522 ts_tablet_manager.cc:1909] T 9babdea7835440909cf3763ca9ffcb64 P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:53.105656 27116 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet e095917962d4431c92c64190c7ddf55b
I20240502 08:17:53.106238 27369 tablet_service.cc:1514] Processing DeleteTablet for tablet e095917962d4431c92c64190c7ddf55b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:53.106995 27236 tablet_service.cc:1514] Processing DeleteTablet for tablet e095917962d4431c92c64190c7ddf55b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:53.107740 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 9babdea7835440909cf3763ca9ffcb64 (table rebalance_test_table_2 [id=1b92061ddf9546eb99771240ff39eac9]) successfully deleted
I20240502 08:17:53.108405 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet e095917962d4431c92c64190c7ddf55b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:53.109800 30522 tablet_replica.cc:330] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:53.110487 30522 raft_consensus.cc:2229] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:53.110973 30522 raft_consensus.cc:2258] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.111830 30522 ts_tablet_manager.cc:1875] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:53.111917 30524 ts_tablet_manager.cc:1888] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:53.112138 30524 log.cc:1196] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/9babdea7835440909cf3763ca9ffcb64
I20240502 08:17:53.112740 30524 ts_tablet_manager.cc:1909] T 9babdea7835440909cf3763ca9ffcb64 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:53.114163 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet 9babdea7835440909cf3763ca9ffcb64 (table rebalance_test_table_2 [id=1b92061ddf9546eb99771240ff39eac9]) successfully deleted
I20240502 08:17:53.114708 30524 tablet_replica.cc:330] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:53.115025 30524 raft_consensus.cc:2229] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:53.115504 30524 raft_consensus.cc:2258] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.116349 30524 ts_tablet_manager.cc:1875] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:53.120671 30521 ts_tablet_manager.cc:1888] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:53.120863 30521 log.cc:1196] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/82d6bd3211e34b2e85231b09b95e1587
I20240502 08:17:53.121397 30521 ts_tablet_manager.cc:1909] T 82d6bd3211e34b2e85231b09b95e1587 P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:17:53.122931 30523 tablet_replica.cc:330] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:53.123036 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 9babdea7835440909cf3763ca9ffcb64 (table rebalance_test_table_2 [id=1b92061ddf9546eb99771240ff39eac9]) successfully deleted
I20240502 08:17:53.123075 30524 ts_tablet_manager.cc:1888] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:53.123204 30524 log.cc:1196] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/e095917962d4431c92c64190c7ddf55b
I20240502 08:17:53.123672 30524 ts_tablet_manager.cc:1909] T e095917962d4431c92c64190c7ddf55b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:53.124547 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet e095917962d4431c92c64190c7ddf55b (table rebalance_test_table_2 [id=1b92061ddf9546eb99771240ff39eac9]) successfully deleted
I20240502 08:17:53.124862 30523 raft_consensus.cc:2229] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:53.126667 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 82d6bd3211e34b2e85231b09b95e1587 (table rebalance_test_table_2 [id=1b92061ddf9546eb99771240ff39eac9]) successfully deleted
I20240502 08:17:53.126821 30523 raft_consensus.cc:2258] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.127285 30522 ts_tablet_manager.cc:1888] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:53.127424 30522 log.cc:1196] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/82d6bd3211e34b2e85231b09b95e1587
I20240502 08:17:53.127569 30523 ts_tablet_manager.cc:1875] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:53.127849 30522 ts_tablet_manager.cc:1909] T 82d6bd3211e34b2e85231b09b95e1587 P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:53.131804 30494 raft_consensus.cc:482] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:53.131983 30494 raft_consensus.cc:504] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:53.133910 30494 leader_election.cc:288] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:17:53.133975 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 82d6bd3211e34b2e85231b09b95e1587 (table rebalance_test_table_2 [id=1b92061ddf9546eb99771240ff39eac9]) successfully deleted
I20240502 08:17:53.135967 30522 tablet_replica.cc:330] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:53.136154 30522 raft_consensus.cc:2229] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:53.136379 30522 raft_consensus.cc:2258] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.140146 30522 ts_tablet_manager.cc:1875] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:53.155889 27236 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:17:53.156476 30522 ts_tablet_manager.cc:1888] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:53.157665 30523 ts_tablet_manager.cc:1888] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:53.158586 30522 log.cc:1196] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/e095917962d4431c92c64190c7ddf55b
I20240502 08:17:53.159193 30522 ts_tablet_manager.cc:1909] T e095917962d4431c92c64190c7ddf55b P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:53.160656 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet e095917962d4431c92c64190c7ddf55b (table rebalance_test_table_2 [id=1b92061ddf9546eb99771240ff39eac9]) successfully deleted
I20240502 08:17:53.162434 30523 log.cc:1196] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/82d6bd3211e34b2e85231b09b95e1587
I20240502 08:17:53.162909 30523 ts_tablet_manager.cc:1909] T 82d6bd3211e34b2e85231b09b95e1587 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:53.171778 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 82d6bd3211e34b2e85231b09b95e1587 (table rebalance_test_table_2 [id=1b92061ddf9546eb99771240ff39eac9]) successfully deleted
I20240502 08:17:53.173619 30523 tablet_replica.cc:330] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:53.174293 30523 raft_consensus.cc:2229] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:53.175613 30523 raft_consensus.cc:2258] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.176522 30523 ts_tablet_manager.cc:1875] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:53.183828 30497 raft_consensus.cc:482] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:53.184026 30497 raft_consensus.cc:504] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:53.184944 30497 leader_election.cc:288] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:53.186391 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d5687f84714fc7a5f3885c8efb272b" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:17:53.186697 30159 raft_consensus.cc:2454] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:17:53.187474 29964 leader_election.cc:302] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:53.187809 30497 raft_consensus.cc:2781] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:53.187884 30497 raft_consensus.cc:482] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:53.187992 30497 raft_consensus.cc:3037] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.190853 30008 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:53.192130 30497 raft_consensus.cc:504] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:53.194216 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d5687f84714fc7a5f3885c8efb272b" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:17:53.194482 30159 raft_consensus.cc:3037] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.197198 30159 raft_consensus.cc:2454] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:17:53.197696 29964 leader_election.cc:302] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:53.198141 30489 raft_consensus.cc:2781] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:53.205435 30497 leader_election.cc:288] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:53.207247 27369 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:17:53.222956 30289 raft_consensus.cc:3037] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.226871 30500 consensus_queue.cc:1035] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:53.233916 30523 ts_tablet_manager.cc:1888] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:53.234192 30523 log.cc:1196] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/e095917962d4431c92c64190c7ddf55b
I20240502 08:17:53.234776 30523 ts_tablet_manager.cc:1909] T e095917962d4431c92c64190c7ddf55b P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:53.236048 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet e095917962d4431c92c64190c7ddf55b (table rebalance_test_table_2 [id=1b92061ddf9546eb99771240ff39eac9]) successfully deleted
I20240502 08:17:53.241554 30556 consensus_queue.cc:1035] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:53.252874 30268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:53.259541 30489 raft_consensus.cc:686] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:17:53.260358 30489 consensus_queue.cc:237] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:53.277631 30402 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:53.296391 27116 catalog_manager.cc:5484] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:53.300707 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d5687f84714fc7a5f3885c8efb272b" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:17:53.302302 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d5687f84714fc7a5f3885c8efb272b" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:17:53.302688 30423 raft_consensus.cc:2454] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:17:53.303107 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d5687f84714fc7a5f3885c8efb272b" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:17:53.303411 30422 raft_consensus.cc:2454] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:17:53.303634 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d5687f84714fc7a5f3885c8efb272b" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:17:53.303891 30421 raft_consensus.cc:3037] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.304180 30093 leader_election.cc:302] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters: 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:17:53.304562 30494 raft_consensus.cc:2741] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240502 08:17:53.310448 30500 consensus_queue.cc:1035] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:53.328792 30421 raft_consensus.cc:2454] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:17:53.416955 30335 consensus_queue.cc:1035] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:53.442111 27116 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:17:53.458889 27499 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:17:53.460335 30159 raft_consensus.cc:1261] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:53.461220 30489 consensus_queue.cc:1035] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:53.461376 30140 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:53.469019 27112 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:53.478096 27631 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:17:53.482748 30140 tablet_service.cc:1467] Processing CreateTablet for tablet 0d33916984034ab99f957026b3e30d38 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e14b3e831e834278ac631fa1cb84dced]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:53.483179 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d33916984034ab99f957026b3e30d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.483624 30402 tablet_service.cc:1467] Processing CreateTablet for tablet dd644c6742854b79a212d858fa9df147 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e14b3e831e834278ac631fa1cb84dced]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:53.484393 30007 tablet_service.cc:1467] Processing CreateTablet for tablet 0d33916984034ab99f957026b3e30d38 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e14b3e831e834278ac631fa1cb84dced]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:53.484815 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d33916984034ab99f957026b3e30d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.488293 30268 tablet_service.cc:1467] Processing CreateTablet for tablet dd644c6742854b79a212d858fa9df147 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e14b3e831e834278ac631fa1cb84dced]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:53.488675 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd644c6742854b79a212d858fa9df147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.489706 30402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd644c6742854b79a212d858fa9df147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.493067 30421 raft_consensus.cc:1261] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:53.496313 30605 consensus_queue.cc:1035] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:53.499210 27112 catalog_manager.cc:5576] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899) does not have the latest schema for tablet b9d5687f84714fc7a5f3885c8efb272b (table rebalancer_dynamic_table [id=f86da6ea782d4a358678424e91733096]). Expected version 1 got 0
I20240502 08:17:53.500741 30623 tablet_bootstrap.cc:492] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:53.502676 30008 tablet_service.cc:1467] Processing CreateTablet for tablet dd644c6742854b79a212d858fa9df147 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e14b3e831e834278ac631fa1cb84dced]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:53.503078 30008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd644c6742854b79a212d858fa9df147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.508240 30624 tablet.cc:1722] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:53.509255 30623 tablet_bootstrap.cc:654] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.509945 27112 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet b9d5687f84714fc7a5f3885c8efb272b (table rebalancer_dynamic_table [id=f86da6ea782d4a358678424e91733096]). Expected version 1 got 0
I20240502 08:17:53.510910 27236 tablet_service.cc:1467] Processing CreateTablet for tablet 08245cda17c44a77bcde2f864d2045ec (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e14b3e831e834278ac631fa1cb84dced]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:53.511372 27236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08245cda17c44a77bcde2f864d2045ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.511356 30270 tablet_service.cc:1467] Processing CreateTablet for tablet 0d33916984034ab99f957026b3e30d38 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e14b3e831e834278ac631fa1cb84dced]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:53.511729 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d33916984034ab99f957026b3e30d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.514091 30624 tablet.cc:1620] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:17:53.514390 30269 tablet_service.cc:1467] Processing CreateTablet for tablet 08245cda17c44a77bcde2f864d2045ec (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e14b3e831e834278ac631fa1cb84dced]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:53.514755 30269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08245cda17c44a77bcde2f864d2045ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.517464 30627 consensus_queue.cc:1035] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:53.523428 30627 consensus_queue.cc:1035] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:53.527348 30401 tablet_service.cc:1467] Processing CreateTablet for tablet 08245cda17c44a77bcde2f864d2045ec (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e14b3e831e834278ac631fa1cb84dced]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:53.527808 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08245cda17c44a77bcde2f864d2045ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.553321 30626 tablet.cc:1722] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:53.554342 30626 tablet.cc:1620] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:17:53.555369 30628 tablet_bootstrap.cc:492] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:17:53.557442 30628 tablet_bootstrap.cc:654] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.558880 30630 tablet_bootstrap.cc:492] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:53.560782 30630 tablet_bootstrap.cc:654] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.563658 27112 catalog_manager.cc:5576] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899) does not have the latest schema for tablet b9d5687f84714fc7a5f3885c8efb272b (table rebalancer_dynamic_table [id=f86da6ea782d4a358678424e91733096]). Expected version 1 got 0
I20240502 08:17:53.565492 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=f86da6ea782d4a358678424e91733096] alter complete (version 1)
I20240502 08:17:53.570581 30634 tablet_bootstrap.cc:492] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:53.572925 30634 tablet_bootstrap.cc:654] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.580509 30630 tablet_bootstrap.cc:492] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:53.580742 30630 ts_tablet_manager.cc:1367] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240502 08:17:53.581583 30630 raft_consensus.cc:348] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.587152 30628 tablet_bootstrap.cc:492] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:17:53.587410 30628 ts_tablet_manager.cc:1367] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20240502 08:17:53.588472 30628 raft_consensus.cc:348] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.592051 30634 tablet_bootstrap.cc:492] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:53.592228 30630 raft_consensus.cc:374] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.595679 30633 tablet_bootstrap.cc:492] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:53.595712 27112 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet b9d5687f84714fc7a5f3885c8efb272b (table rebalancer_dynamic_table [id=f86da6ea782d4a358678424e91733096]). Expected version 1 got 0
I20240502 08:17:53.601244 30630 raft_consensus.cc:729] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.603924 30630 consensus_queue.cc:260] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.604386 30628 raft_consensus.cc:374] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.604543 30628 raft_consensus.cc:729] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.604872 30628 consensus_queue.cc:260] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.605700 30628 ts_tablet_manager.cc:1398] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240502 08:17:53.606112 30628 tablet_bootstrap.cc:492] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:17:53.607813 30628 tablet_bootstrap.cc:654] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.602581 30639 mvcc.cc:205] Tried to move back new op lower bound from 7023156729678000128 to 7023156728879357952. Current Snapshot: MvccSnapshot[applied={T|T < 7023156729678000128}]
I20240502 08:17:53.608561 30634 ts_tablet_manager.cc:1367] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.000s
I20240502 08:17:53.609668 30634 raft_consensus.cc:348] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.610065 30634 raft_consensus.cc:374] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.610231 30634 raft_consensus.cc:729] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.610350 27112 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet b9d5687f84714fc7a5f3885c8efb272b (table rebalancer_dynamic_table [id=f86da6ea782d4a358678424e91733096]). Expected version 1 got 0
I20240502 08:17:53.610881 30634 consensus_queue.cc:260] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.613029 30633 tablet_bootstrap.cc:654] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.614517 27112 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 08:17:53.614727 27112 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 08:17:53.615377 30630 ts_tablet_manager.cc:1398] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20240502 08:17:53.620050 27112 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 31edd3f3b1ec4b99b9bc7e16762e9911
I20240502 08:17:53.608770 30642 tablet.cc:1722] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:53.621315 27112 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 515abcc30e81445dae3103a40045691b
I20240502 08:17:53.621414 27236 tablet_service.cc:1514] Processing DeleteTablet for tablet 31edd3f3b1ec4b99b9bc7e16762e9911 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:53.621670 30524 tablet_replica.cc:330] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:53.621881 30524 raft_consensus.cc:2229] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:53.621976 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet 515abcc30e81445dae3103a40045691b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:53.622247 30524 raft_consensus.cc:2258] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.622303 30522 tablet_replica.cc:330] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:53.622519 30522 raft_consensus.cc:2229] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:53.622874 27112 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet f0332286514c49e2812434ec7043cb90
I20240502 08:17:53.622931 30522 raft_consensus.cc:2258] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.622980 30524 ts_tablet_manager.cc:1875] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:53.623570 30522 ts_tablet_manager.cc:1875] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:53.623756 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet f0332286514c49e2812434ec7043cb90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:17:53.624004 27499 tablet_service.cc:1514] Processing DeleteTablet for tablet 31edd3f3b1ec4b99b9bc7e16762e9911 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:17:53.624215 30521 tablet_replica.cc:330] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:17:53.624408 30521 raft_consensus.cc:2229] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:53.624475 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet f0332286514c49e2812434ec7043cb90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:53.624692 30521 raft_consensus.cc:2258] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.625370 30521 ts_tablet_manager.cc:1875] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:53.625618 27236 tablet_service.cc:1514] Processing DeleteTablet for tablet 515abcc30e81445dae3103a40045691b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:53.625679 30623 tablet_bootstrap.cc:492] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:17:53.625813 30623 ts_tablet_manager.cc:1367] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.125s user 0.035s sys 0.032s
I20240502 08:17:53.626624 30623 raft_consensus.cc:348] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.626838 30623 raft_consensus.cc:374] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.626920 30623 raft_consensus.cc:729] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.627260 30623 consensus_queue.cc:260] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.627506 27369 tablet_service.cc:1514] Processing DeleteTablet for tablet 31edd3f3b1ec4b99b9bc7e16762e9911 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:53.627704 27370 tablet_service.cc:1514] Processing DeleteTablet for tablet 515abcc30e81445dae3103a40045691b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:53.627786 30523 tablet_replica.cc:330] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:53.627828 30623 ts_tablet_manager.cc:1398] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:53.628034 30523 raft_consensus.cc:2229] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:53.628072 27368 tablet_service.cc:1514] Processing DeleteTablet for tablet f0332286514c49e2812434ec7043cb90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:53.628293 30523 raft_consensus.cc:2258] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.628973 30523 ts_tablet_manager.cc:1875] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:53.631585 30524 ts_tablet_manager.cc:1888] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:53.631767 30524 log.cc:1196] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/31edd3f3b1ec4b99b9bc7e16762e9911
I20240502 08:17:53.632162 30524 ts_tablet_manager.cc:1909] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:53.632882 30524 tablet_replica.cc:330] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:53.633003 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet 31edd3f3b1ec4b99b9bc7e16762e9911 (table rebalance_test_table_4 [id=6da106d4ea7c4b0c887fce2b340247eb]) successfully deleted
I20240502 08:17:53.633229 30524 raft_consensus.cc:2229] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:53.633412 30524 raft_consensus.cc:2258] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.633754 30634 ts_tablet_manager.cc:1398] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240502 08:17:53.634502 30524 ts_tablet_manager.cc:1875] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:53.636503 30634 tablet_bootstrap.cc:492] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:53.637344 30522 ts_tablet_manager.cc:1888] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:53.638303 30523 ts_tablet_manager.cc:1888] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:53.638495 30523 log.cc:1196] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/31edd3f3b1ec4b99b9bc7e16762e9911
I20240502 08:17:53.639009 30523 ts_tablet_manager.cc:1909] T 31edd3f3b1ec4b99b9bc7e16762e9911 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:53.639182 30522 log.cc:1196] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/515abcc30e81445dae3103a40045691b
I20240502 08:17:53.639613 30522 ts_tablet_manager.cc:1909] T 515abcc30e81445dae3103a40045691b P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:53.640548 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 31edd3f3b1ec4b99b9bc7e16762e9911 (table rebalance_test_table_4 [id=6da106d4ea7c4b0c887fce2b340247eb]) successfully deleted
I20240502 08:17:53.641356 30522 tablet_replica.cc:330] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:53.641455 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 515abcc30e81445dae3103a40045691b (table rebalance_test_table_4 [id=6da106d4ea7c4b0c887fce2b340247eb]) successfully deleted
I20240502 08:17:53.641624 30523 tablet_replica.cc:330] T 515abcc30e81445dae3103a40045691b P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:53.641835 30523 raft_consensus.cc:2229] T 515abcc30e81445dae3103a40045691b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:53.642123 30523 raft_consensus.cc:2258] T 515abcc30e81445dae3103a40045691b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.642526 30634 tablet_bootstrap.cc:654] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.642660 30521 ts_tablet_manager.cc:1888] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:53.642803 30522 raft_consensus.cc:2229] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:53.642886 30521 log.cc:1196] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/f0332286514c49e2812434ec7043cb90
I20240502 08:17:53.643358 30521 ts_tablet_manager.cc:1909] T f0332286514c49e2812434ec7043cb90 P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:17:53.643715 30522 raft_consensus.cc:2258] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.644300 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet f0332286514c49e2812434ec7043cb90 (table rebalance_test_table_4 [id=6da106d4ea7c4b0c887fce2b340247eb]) successfully deleted
I20240502 08:17:53.645450 30522 ts_tablet_manager.cc:1875] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:53.646754 30642 tablet.cc:1620] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:17:53.646960 30523 ts_tablet_manager.cc:1875] T 515abcc30e81445dae3103a40045691b P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:53.647533 30521 tablet_replica.cc:330] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:17:53.647738 30521 raft_consensus.cc:2229] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:53.648233 30521 raft_consensus.cc:2258] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.648910 30521 ts_tablet_manager.cc:1875] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:53.652573 30524 ts_tablet_manager.cc:1888] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:53.653738 30524 log.cc:1196] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/515abcc30e81445dae3103a40045691b
I20240502 08:17:53.654520 30524 ts_tablet_manager.cc:1909] T 515abcc30e81445dae3103a40045691b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:53.656323 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet 515abcc30e81445dae3103a40045691b (table rebalance_test_table_4 [id=6da106d4ea7c4b0c887fce2b340247eb]) successfully deleted
I20240502 08:17:53.661266 30633 tablet_bootstrap.cc:492] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:53.661417 30633 ts_tablet_manager.cc:1367] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.000s
I20240502 08:17:53.662266 30633 raft_consensus.cc:348] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.662442 30633 raft_consensus.cc:374] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.662505 30633 raft_consensus.cc:729] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.662701 30633 consensus_queue.cc:260] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
W20240502 08:17:53.667209 27304 tablet.cc:2369] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:53.667582 30633 ts_tablet_manager.cc:1398] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:17:53.670110 30633 tablet_bootstrap.cc:492] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:53.671620 30633 tablet_bootstrap.cc:654] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.672245 30523 ts_tablet_manager.cc:1888] T 515abcc30e81445dae3103a40045691b P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:53.672385 30523 log.cc:1196] T 515abcc30e81445dae3103a40045691b P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/515abcc30e81445dae3103a40045691b
I20240502 08:17:53.672827 30523 ts_tablet_manager.cc:1909] T 515abcc30e81445dae3103a40045691b P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:53.673854 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 515abcc30e81445dae3103a40045691b (table rebalance_test_table_4 [id=6da106d4ea7c4b0c887fce2b340247eb]) successfully deleted
I20240502 08:17:53.676558 30521 ts_tablet_manager.cc:1888] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:53.683852 30522 ts_tablet_manager.cc:1888] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:53.685053 30634 tablet_bootstrap.cc:492] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:53.685297 30634 ts_tablet_manager.cc:1367] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.049s user 0.011s sys 0.023s
I20240502 08:17:53.685235 27112 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:17:53.685454 30522 log.cc:1196] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/f0332286514c49e2812434ec7043cb90
I20240502 08:17:53.686486 30522 ts_tablet_manager.cc:1909] T f0332286514c49e2812434ec7043cb90 P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:53.686794 30628 tablet_bootstrap.cc:492] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:17:53.687040 30628 ts_tablet_manager.cc:1367] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.081s user 0.013s sys 0.001s
I20240502 08:17:53.687320 30634 raft_consensus.cc:348] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.688241 30634 raft_consensus.cc:374] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.688431 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet f0332286514c49e2812434ec7043cb90 (table rebalance_test_table_4 [id=6da106d4ea7c4b0c887fce2b340247eb]) successfully deleted
I20240502 08:17:53.687781 30628 raft_consensus.cc:348] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.688591 30628 raft_consensus.cc:374] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.688663 30628 raft_consensus.cc:729] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.688933 30628 consensus_queue.cc:260] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.689164 30634 raft_consensus.cc:729] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.689308 30523 tablet_replica.cc:330] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:53.689541 30628 ts_tablet_manager.cc:1398] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:53.689862 30523 raft_consensus.cc:2229] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:53.689554 30634 consensus_queue.cc:260] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.694298 30521 log.cc:1196] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/31edd3f3b1ec4b99b9bc7e16762e9911
I20240502 08:17:53.694298 30523 raft_consensus.cc:2258] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.695477 30634 ts_tablet_manager.cc:1398] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 08:17:53.695801 30634 tablet_bootstrap.cc:492] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:53.696012 30633 tablet_bootstrap.cc:492] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:53.696192 30633 ts_tablet_manager.cc:1367] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240502 08:17:53.696929 30633 raft_consensus.cc:348] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.697165 30633 raft_consensus.cc:374] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.697239 30633 raft_consensus.cc:729] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.697408 30634 tablet_bootstrap.cc:654] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.697515 30633 consensus_queue.cc:260] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.698698 30633 ts_tablet_manager.cc:1398] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:53.698902 30624 tablet.cc:1722] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:53.699368 30624 tablet.cc:1620] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:17:53.701018 30523 ts_tablet_manager.cc:1875] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:53.702814 30626 tablet.cc:1722] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:53.703220 30626 tablet.cc:1620] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:17:53.705410 30642 tablet.cc:1722] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:53.705937 30642 tablet.cc:1620] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:17:53.712270 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=f86da6ea782d4a358678424e91733096] alter complete (version 2)
I20240502 08:17:53.715210 30523 ts_tablet_manager.cc:1888] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:53.715350 30523 log.cc:1196] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/f0332286514c49e2812434ec7043cb90
I20240502 08:17:53.715730 30523 ts_tablet_manager.cc:1909] T f0332286514c49e2812434ec7043cb90 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:53.716656 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet f0332286514c49e2812434ec7043cb90 (table rebalance_test_table_4 [id=6da106d4ea7c4b0c887fce2b340247eb]) successfully deleted
I20240502 08:17:53.718258 30521 ts_tablet_manager.cc:1909] T 31edd3f3b1ec4b99b9bc7e16762e9911 P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:17:53.720135 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 31edd3f3b1ec4b99b9bc7e16762e9911 (table rebalance_test_table_4 [id=6da106d4ea7c4b0c887fce2b340247eb]) successfully deleted
I20240502 08:17:53.721613 30634 tablet_bootstrap.cc:492] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:53.721805 30634 ts_tablet_manager.cc:1367] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.001s
I20240502 08:17:53.722608 30634 raft_consensus.cc:348] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.722878 30634 raft_consensus.cc:374] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.722995 30634 raft_consensus.cc:729] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.723255 30634 consensus_queue.cc:260] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.724543 30634 ts_tablet_manager.cc:1398] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:17:53.738833 30545 raft_consensus.cc:482] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:53.738963 30545 raft_consensus.cc:504] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.740594 30545 leader_election.cc:288] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:17:53.766938 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd644c6742854b79a212d858fa9df147" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:17:53.767160 30421 raft_consensus.cc:2454] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:17:53.767783 30223 leader_election.cc:302] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:17:53.768221 30545 raft_consensus.cc:2781] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:53.768354 30545 raft_consensus.cc:482] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:53.768432 30545 raft_consensus.cc:3037] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.771476 30545 raft_consensus.cc:504] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.772190 30545 leader_election.cc:288] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:17:53.775197 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd644c6742854b79a212d858fa9df147" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:17:53.775388 30421 raft_consensus.cc:3037] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.781848 30421 raft_consensus.cc:2454] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:17:53.783375 30223 leader_election.cc:302] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:17:53.783749 30545 raft_consensus.cc:2781] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:53.783661 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd644c6742854b79a212d858fa9df147" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:17:53.783986 30028 raft_consensus.cc:2454] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:17:53.784313 30545 raft_consensus.cc:686] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:17:53.784359 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd644c6742854b79a212d858fa9df147" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:17:53.784639 30027 raft_consensus.cc:3037] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.784801 30545 consensus_queue.cc:237] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.788476 30027 raft_consensus.cc:2454] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:17:53.796731 27110 catalog_manager.cc:5484] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7). New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:53.802606 30494 raft_consensus.cc:482] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:53.802784 30494 raft_consensus.cc:504] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.803627 30494 leader_election.cc:288] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:17:53.804406 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d33916984034ab99f957026b3e30d38" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:17:53.804630 30027 raft_consensus.cc:2454] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:17:53.805303 30093 leader_election.cc:302] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:53.805601 30494 raft_consensus.cc:2781] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:53.805711 30494 raft_consensus.cc:482] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:53.805809 30494 raft_consensus.cc:3037] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.808632 30494 raft_consensus.cc:504] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.812175 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d33916984034ab99f957026b3e30d38" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:17:53.812388 30027 raft_consensus.cc:3037] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.814965 30027 raft_consensus.cc:2454] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:17:53.816562 30093 leader_election.cc:302] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:53.817255 30667 raft_consensus.cc:2781] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:53.817818 30667 raft_consensus.cc:686] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:17:53.817960 30494 leader_election.cc:288] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:17:53.818416 30667 consensus_queue.cc:237] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.818809 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d33916984034ab99f957026b3e30d38" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:17:53.819064 30288 raft_consensus.cc:2454] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:17:53.819227 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:17:53.819406 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d33916984034ab99f957026b3e30d38" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:17:53.819592 30290 raft_consensus.cc:3037] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.822154 30290 raft_consensus.cc:2454] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:17:53.823984 27114 catalog_manager.cc:5484] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:53.827791 30624 tablet.cc:1722] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:53.829387 30642 tablet.cc:1722] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:53.836541 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=f86da6ea782d4a358678424e91733096] alter complete (version 3)
I20240502 08:17:53.837410 30626 tablet.cc:1722] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:53.857704 30545 raft_consensus.cc:482] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:53.857877 30545 raft_consensus.cc:504] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.858589 30545 leader_election.cc:288] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:53.858983 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08245cda17c44a77bcde2f864d2045ec" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:17:53.859215 30421 raft_consensus.cc:2454] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:17:53.859634 30223 leader_election.cc:302] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:17:53.859916 30545 raft_consensus.cc:2781] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:53.860006 30545 raft_consensus.cc:482] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:53.860095 30545 raft_consensus.cc:3037] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.862702 30545 raft_consensus.cc:504] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.863340 30545 leader_election.cc:288] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:53.863701 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08245cda17c44a77bcde2f864d2045ec" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:17:53.863899 30421 raft_consensus.cc:3037] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.865978 27256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08245cda17c44a77bcde2f864d2045ec" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:17:53.866250 27256 raft_consensus.cc:2454] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:17:53.866542 30421 raft_consensus.cc:2454] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:17:53.866848 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08245cda17c44a77bcde2f864d2045ec" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:17:53.866909 30223 leader_election.cc:302] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:17:53.867084 27258 raft_consensus.cc:3037] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.867158 30545 raft_consensus.cc:2781] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:53.867311 30545 raft_consensus.cc:686] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:17:53.867611 30545 consensus_queue.cc:237] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:53.869233 27258 raft_consensus.cc:2454] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:17:53.870260 27110 catalog_manager.cc:5484] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7). New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:53.943681 27110 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:17:53.944005 27110 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:17:53.948124 27110 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet b9d5687f84714fc7a5f3885c8efb272b
I20240502 08:17:53.949110 30402 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d5687f84714fc7a5f3885c8efb272b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:56694
I20240502 08:17:53.950136 30008 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d5687f84714fc7a5f3885c8efb272b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:58374
I20240502 08:17:53.950594 30140 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d5687f84714fc7a5f3885c8efb272b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:53 UTC) from {username='slave'} at 127.0.0.1:48808
I20240502 08:17:53.988464 30703 tablet_replica.cc:330] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:17:53.988687 30703 raft_consensus.cc:2229] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:53.988895 30703 raft_consensus.cc:2258] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.990499 30702 tablet_replica.cc:330] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:17:53.990711 30702 raft_consensus.cc:2229] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:53.991080 30702 raft_consensus.cc:2258] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.991689 30701 tablet_replica.cc:330] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:17:53.992158 30701 raft_consensus.cc:2229] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:53.992524 30701 raft_consensus.cc:2258] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.994143 30701 ts_tablet_manager.cc:1875] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.005659 30702 ts_tablet_manager.cc:1875] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.009467 30701 ts_tablet_manager.cc:1888] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:54.009742 30701 log.cc:1196] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/b9d5687f84714fc7a5f3885c8efb272b
I20240502 08:17:54.010439 30701 ts_tablet_manager.cc:1909] T b9d5687f84714fc7a5f3885c8efb272b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting consensus metadata
I20240502 08:17:54.010483 30703 ts_tablet_manager.cc:1875] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.014360 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet b9d5687f84714fc7a5f3885c8efb272b (table rebalancer_dynamic_table_new_name [id=f86da6ea782d4a358678424e91733096]) successfully deleted
I20240502 08:17:54.027504 30702 ts_tablet_manager.cc:1888] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:54.027757 30702 log.cc:1196] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/b9d5687f84714fc7a5f3885c8efb272b
I20240502 08:17:54.028348 30702 ts_tablet_manager.cc:1909] T b9d5687f84714fc7a5f3885c8efb272b P 1f0bf4bc3c5146539146b825b048bff3: Deleting consensus metadata
I20240502 08:17:54.030328 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet b9d5687f84714fc7a5f3885c8efb272b (table rebalancer_dynamic_table_new_name [id=f86da6ea782d4a358678424e91733096]) successfully deleted
I20240502 08:17:54.030596 30703 ts_tablet_manager.cc:1888] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:54.030716 30703 log.cc:1196] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/b9d5687f84714fc7a5f3885c8efb272b
I20240502 08:17:54.031215 30703 ts_tablet_manager.cc:1909] T b9d5687f84714fc7a5f3885c8efb272b P 84cda20738d2452da166e31fc6da57a6: Deleting consensus metadata
I20240502 08:17:54.032217 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet b9d5687f84714fc7a5f3885c8efb272b (table rebalancer_dynamic_table_new_name [id=f86da6ea782d4a358678424e91733096]) successfully deleted
I20240502 08:17:54.052254 27110 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:17:54.059142 30402 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:54.070241 30008 tablet_service.cc:1467] Processing CreateTablet for tablet 9ffcfdb358274bb680fbb8afae05691a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9bc7251a63034f0bb05e56e417d7dfa8]), partition=
I20240502 08:17:54.070627 30008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ffcfdb358274bb680fbb8afae05691a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.070674 30402 tablet_service.cc:1467] Processing CreateTablet for tablet 9ffcfdb358274bb680fbb8afae05691a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9bc7251a63034f0bb05e56e417d7dfa8]), partition=
I20240502 08:17:54.071074 30402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ffcfdb358274bb680fbb8afae05691a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.076925 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 9ffcfdb358274bb680fbb8afae05691a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9bc7251a63034f0bb05e56e417d7dfa8]), partition=
I20240502 08:17:54.077316 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ffcfdb358274bb680fbb8afae05691a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.079052 30628 tablet_bootstrap.cc:492] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:17:54.080741 30628 tablet_bootstrap.cc:654] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.081853 30633 tablet_bootstrap.cc:492] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:54.083326 30633 tablet_bootstrap.cc:654] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.084420 30007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:54.093094 30633 tablet_bootstrap.cc:492] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:54.093385 30633 ts_tablet_manager.cc:1367] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 08:17:54.094610 30633 raft_consensus.cc:348] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:54.095036 30633 raft_consensus.cc:374] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.095235 30633 raft_consensus.cc:729] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.095705 30628 tablet_bootstrap.cc:492] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:17:54.095911 30628 ts_tablet_manager.cc:1367] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.007s
I20240502 08:17:54.096107 30633 consensus_queue.cc:260] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:54.097855 30633 ts_tablet_manager.cc:1398] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:54.097460 30628 raft_consensus.cc:348] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:54.098701 30268 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:54.098712 30628 raft_consensus.cc:374] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.098809 30628 raft_consensus.cc:729] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.099056 30628 consensus_queue.cc:260] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:54.099750 30628 ts_tablet_manager.cc:1398] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:17:54.100766 30723 tablet_bootstrap.cc:492] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:54.117785 27236 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:17:54.118371 27368 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:17:54.125806 27110 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 08:17:54.126304 27110 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 08:17:54.131075 27110 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 0866b26202c14efca9a753fa1d2e8071
I20240502 08:17:54.131532 27110 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 72eb1c4f44934edfb49a3d6a39618df4
I20240502 08:17:54.131891 27110 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 0662521f50104435aedf647a56f57c33
I20240502 08:17:54.133301 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet 0866b26202c14efca9a753fa1d2e8071 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:54.133493 27369 tablet_service.cc:1514] Processing DeleteTablet for tablet 0662521f50104435aedf647a56f57c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:54.133527 27629 tablet_service.cc:1514] Processing DeleteTablet for tablet 72eb1c4f44934edfb49a3d6a39618df4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:54.133714 27630 tablet_service.cc:1514] Processing DeleteTablet for tablet 0662521f50104435aedf647a56f57c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:54.133764 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet 0662521f50104435aedf647a56f57c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:54.133919 30522 tablet_replica.cc:330] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:54.134063 30523 tablet_replica.cc:330] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:54.134047 27236 tablet_service.cc:1514] Processing DeleteTablet for tablet 0866b26202c14efca9a753fa1d2e8071 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:54.134160 30524 tablet_replica.cc:330] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:54.134158 30522 raft_consensus.cc:2229] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:54.133911 27370 tablet_service.cc:1514] Processing DeleteTablet for tablet 72eb1c4f44934edfb49a3d6a39618df4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:54.134258 30523 raft_consensus.cc:2229] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:54.134361 30524 raft_consensus.cc:2229] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:54.134385 30522 raft_consensus.cc:2258] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.134464 30523 raft_consensus.cc:2258] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.134680 30524 raft_consensus.cc:2258] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.135066 30523 ts_tablet_manager.cc:1875] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.135073 30522 ts_tablet_manager.cc:1875] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.135380 30524 ts_tablet_manager.cc:1875] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.133888 27368 tablet_service.cc:1514] Processing DeleteTablet for tablet 0866b26202c14efca9a753fa1d2e8071 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:54.143342 30524 ts_tablet_manager.cc:1888] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:54.143541 30524 log.cc:1196] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/0662521f50104435aedf647a56f57c33
I20240502 08:17:54.144035 30524 ts_tablet_manager.cc:1909] T 0662521f50104435aedf647a56f57c33 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:54.144029 30523 ts_tablet_manager.cc:1888] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:54.144135 30523 log.cc:1196] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/0662521f50104435aedf647a56f57c33
I20240502 08:17:54.144563 30522 ts_tablet_manager.cc:1888] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:54.144670 30523 ts_tablet_manager.cc:1909] T 0662521f50104435aedf647a56f57c33 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:54.146103 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 0662521f50104435aedf647a56f57c33 (table rebalance_test_table_6 [id=9db3ba2ceb8c4fc8835b074aa27409c0]) successfully deleted
I20240502 08:17:54.146564 30522 log.cc:1196] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/0866b26202c14efca9a753fa1d2e8071
I20240502 08:17:54.146692 30523 tablet_replica.cc:330] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:54.146689 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet 0662521f50104435aedf647a56f57c33 (table rebalance_test_table_6 [id=9db3ba2ceb8c4fc8835b074aa27409c0]) successfully deleted
I20240502 08:17:54.146945 30523 raft_consensus.cc:2229] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:54.147261 30523 raft_consensus.cc:2258] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.147815 30522 ts_tablet_manager.cc:1909] T 0866b26202c14efca9a753fa1d2e8071 P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:54.148058 30523 ts_tablet_manager.cc:1875] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.150077 30524 tablet_replica.cc:330] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:54.150476 30723 tablet_bootstrap.cc:654] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.155350 30524 raft_consensus.cc:2229] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:54.156706 30524 raft_consensus.cc:2258] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.157533 30524 ts_tablet_manager.cc:1875] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.159834 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 0866b26202c14efca9a753fa1d2e8071 (table rebalance_test_table_6 [id=9db3ba2ceb8c4fc8835b074aa27409c0]) successfully deleted
I20240502 08:17:54.166257 30522 tablet_replica.cc:330] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:54.169509 30522 raft_consensus.cc:2229] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:54.173069 30140 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:54.170643 30522 raft_consensus.cc:2258] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.173756 30522 ts_tablet_manager.cc:1875] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.187542 27499 tablet_service.cc:1514] Processing DeleteTablet for tablet 72eb1c4f44934edfb49a3d6a39618df4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:17:54.187790 30521 tablet_replica.cc:330] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:17:54.187978 30521 raft_consensus.cc:2229] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:54.188212 30521 raft_consensus.cc:2258] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.188882 30521 ts_tablet_manager.cc:1875] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.195901 30521 ts_tablet_manager.cc:1888] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:54.196110 30521 log.cc:1196] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/72eb1c4f44934edfb49a3d6a39618df4
I20240502 08:17:54.196554 30521 ts_tablet_manager.cc:1909] T 72eb1c4f44934edfb49a3d6a39618df4 P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:17:54.197530 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 72eb1c4f44934edfb49a3d6a39618df4 (table rebalance_test_table_6 [id=9db3ba2ceb8c4fc8835b074aa27409c0]) successfully deleted
I20240502 08:17:54.204833 30524 ts_tablet_manager.cc:1888] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:54.204998 30524 log.cc:1196] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/0866b26202c14efca9a753fa1d2e8071
I20240502 08:17:54.207032 30723 tablet_bootstrap.cc:492] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:54.207238 30723 ts_tablet_manager.cc:1367] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.107s user 0.000s sys 0.007s
I20240502 08:17:54.208300 30723 raft_consensus.cc:348] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:54.208575 30723 raft_consensus.cc:374] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.208694 30723 raft_consensus.cc:729] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.208946 30723 consensus_queue.cc:260] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:54.209663 30545 consensus_queue.cc:1035] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:54.210877 30723 ts_tablet_manager.cc:1398] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:17:54.211258 30524 ts_tablet_manager.cc:1909] T 0866b26202c14efca9a753fa1d2e8071 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:54.212410 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet 0866b26202c14efca9a753fa1d2e8071 (table rebalance_test_table_6 [id=9db3ba2ceb8c4fc8835b074aa27409c0]) successfully deleted
I20240502 08:17:54.214062 27630 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:17:54.222806 30522 ts_tablet_manager.cc:1888] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:54.222991 30522 log.cc:1196] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/72eb1c4f44934edfb49a3d6a39618df4
I20240502 08:17:54.223552 30522 ts_tablet_manager.cc:1909] T 72eb1c4f44934edfb49a3d6a39618df4 P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:54.225838 30522 tablet_replica.cc:330] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:54.225920 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 72eb1c4f44934edfb49a3d6a39618df4 (table rebalance_test_table_6 [id=9db3ba2ceb8c4fc8835b074aa27409c0]) successfully deleted
I20240502 08:17:54.226197 30522 raft_consensus.cc:2229] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:54.231768 30522 raft_consensus.cc:2258] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.232744 30522 ts_tablet_manager.cc:1875] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.239001 30523 ts_tablet_manager.cc:1888] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:54.242276 30523 log.cc:1196] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/72eb1c4f44934edfb49a3d6a39618df4
I20240502 08:17:54.242866 30523 ts_tablet_manager.cc:1909] T 72eb1c4f44934edfb49a3d6a39618df4 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:54.243839 30627 raft_consensus.cc:482] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:54.244002 30627 raft_consensus.cc:504] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:54.244949 30627 leader_election.cc:288] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:54.245393 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ffcfdb358274bb680fbb8afae05691a" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:17:54.245625 30027 raft_consensus.cc:2454] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:17:54.246100 30355 leader_election.cc:302] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters:
I20240502 08:17:54.246598 30627 raft_consensus.cc:2781] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:54.246691 30627 raft_consensus.cc:482] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:54.246758 30627 raft_consensus.cc:3037] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.248505 30667 consensus_queue.cc:1035] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:54.249415 30627 raft_consensus.cc:504] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:54.249718 27499 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:17:54.250440 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ffcfdb358274bb680fbb8afae05691a" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:17:54.250620 30028 raft_consensus.cc:3037] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.253229 30028 raft_consensus.cc:2454] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:17:54.253679 30355 leader_election.cc:302] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters:
I20240502 08:17:54.254709 30627 leader_election.cc:288] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:54.255134 30627 raft_consensus.cc:2781] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:54.255445 30627 raft_consensus.cc:686] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:17:54.255801 30627 consensus_queue.cc:237] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:54.257181 30653 consensus_queue.cc:1035] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:54.258296 30522 ts_tablet_manager.cc:1888] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:54.266117 30522 log.cc:1196] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/0662521f50104435aedf647a56f57c33
I20240502 08:17:54.266657 30522 ts_tablet_manager.cc:1909] T 0662521f50104435aedf647a56f57c33 P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:54.268450 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 0662521f50104435aedf647a56f57c33 (table rebalance_test_table_6 [id=9db3ba2ceb8c4fc8835b074aa27409c0]) successfully deleted
I20240502 08:17:54.278292 27110 catalog_manager.cc:5484] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:54.281263 30667 consensus_queue.cc:1035] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:54.288878 30653 consensus_queue.cc:1035] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:54.289125 30523 tablet_replica.cc:330] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:54.289176 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 72eb1c4f44934edfb49a3d6a39618df4 (table rebalance_test_table_6 [id=9db3ba2ceb8c4fc8835b074aa27409c0]) successfully deleted
I20240502 08:17:54.289427 30523 raft_consensus.cc:2229] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:54.294049 30523 raft_consensus.cc:2258] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.297333 30523 ts_tablet_manager.cc:1875] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.312129 30523 ts_tablet_manager.cc:1888] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:54.312274 30523 log.cc:1196] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/0866b26202c14efca9a753fa1d2e8071
I20240502 08:17:54.312793 30523 ts_tablet_manager.cc:1909] T 0866b26202c14efca9a753fa1d2e8071 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:54.314410 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 0866b26202c14efca9a753fa1d2e8071 (table rebalance_test_table_6 [id=9db3ba2ceb8c4fc8835b074aa27409c0]) successfully deleted
W20240502 08:17:54.315312 27698 tablet.cc:2369] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:54.386382 30738 consensus_queue.cc:1035] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:54.400281 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ffcfdb358274bb680fbb8afae05691a" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:54.400501 27650 raft_consensus.cc:3037] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.403241 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ffcfdb358274bb680fbb8afae05691a" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:54.405398 27650 raft_consensus.cc:2454] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:17:54.432603 27116 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:17:54.439949 30028 raft_consensus.cc:1261] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:54.440220 27650 raft_consensus.cc:1261] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:54.440567 30627 consensus_queue.cc:1035] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:54.446730 30627 consensus_queue.cc:1035] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:54.448906 27113 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:54.453153 27116 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet 9ffcfdb358274bb680fbb8afae05691a (table rebalancer_dynamic_table [id=9bc7251a63034f0bb05e56e417d7dfa8]). Expected version 1 got 0
I20240502 08:17:54.458837 27368 tablet_service.cc:1467] Processing CreateTablet for tablet 168e71e71b0340fd953d5644b8b8c028 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=767cc24e34dd43bc8e4de07a278f8b27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:54.459440 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 168e71e71b0340fd953d5644b8b8c028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.461421 30747 tablet.cc:1722] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:54.461952 30747 tablet.cc:1620] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:17:54.463863 30746 tablet.cc:1722] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:54.464160 27116 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet 9ffcfdb358274bb680fbb8afae05691a (table rebalancer_dynamic_table [id=9bc7251a63034f0bb05e56e417d7dfa8]). Expected version 1 got 0
I20240502 08:17:54.467933 30140 tablet_service.cc:1467] Processing CreateTablet for tablet 168e71e71b0340fd953d5644b8b8c028 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=767cc24e34dd43bc8e4de07a278f8b27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:54.468053 30402 tablet_service.cc:1467] Processing CreateTablet for tablet 168e71e71b0340fd953d5644b8b8c028 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=767cc24e34dd43bc8e4de07a278f8b27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:54.468240 30746 tablet.cc:1620] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:17:54.468430 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 168e71e71b0340fd953d5644b8b8c028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.468494 30402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 168e71e71b0340fd953d5644b8b8c028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.469347 30139 tablet_service.cc:1467] Processing CreateTablet for tablet e67c99e57a73443da13527ed33326a4e (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=767cc24e34dd43bc8e4de07a278f8b27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:54.469769 30139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e67c99e57a73443da13527ed33326a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.470373 27236 tablet_service.cc:1467] Processing CreateTablet for tablet e67c99e57a73443da13527ed33326a4e (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=767cc24e34dd43bc8e4de07a278f8b27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:54.470788 27236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e67c99e57a73443da13527ed33326a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.474819 27370 tablet_service.cc:1467] Processing CreateTablet for tablet b6a26e01d42b429694641f0ceca11265 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=767cc24e34dd43bc8e4de07a278f8b27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:54.475294 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a26e01d42b429694641f0ceca11265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.477489 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=9bc7251a63034f0bb05e56e417d7dfa8] alter complete (version 1)
I20240502 08:17:54.479372 30268 tablet_service.cc:1467] Processing CreateTablet for tablet e67c99e57a73443da13527ed33326a4e (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=767cc24e34dd43bc8e4de07a278f8b27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:54.479835 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e67c99e57a73443da13527ed33326a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.480151 30007 tablet_service.cc:1467] Processing CreateTablet for tablet b6a26e01d42b429694641f0ceca11265 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=767cc24e34dd43bc8e4de07a278f8b27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:54.480669 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a26e01d42b429694641f0ceca11265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.485828 30749 tablet_bootstrap.cc:492] T 168e71e71b0340fd953d5644b8b8c028 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:54.488124 30749 tablet_bootstrap.cc:654] T 168e71e71b0340fd953d5644b8b8c028 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.488999 30633 tablet_bootstrap.cc:492] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:54.490604 30633 tablet_bootstrap.cc:654] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.497884 30749 tablet_bootstrap.cc:492] T 168e71e71b0340fd953d5644b8b8c028 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:54.498092 30749 ts_tablet_manager.cc:1367] T 168e71e71b0340fd953d5644b8b8c028 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240502 08:17:54.499393 30749 raft_consensus.cc:348] T 168e71e71b0340fd953d5644b8b8c028 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:54.499729 30749 raft_consensus.cc:374] T 168e71e71b0340fd953d5644b8b8c028 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.499878 30749 raft_consensus.cc:729] T 168e71e71b0340fd953d5644b8b8c028 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.500162 30749 consensus_queue.cc:260] T 168e71e71b0340fd953d5644b8b8c028 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:54.501281 30401 tablet_service.cc:1467] Processing CreateTablet for tablet b6a26e01d42b429694641f0ceca11265 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=767cc24e34dd43bc8e4de07a278f8b27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:54.501677 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a26e01d42b429694641f0ceca11265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.502470 30751 tablet_bootstrap.cc:492] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:54.503787 30749 ts_tablet_manager.cc:1398] T 168e71e71b0340fd953d5644b8b8c028 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:17:54.504174 30751 tablet_bootstrap.cc:654] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.504942 30633 tablet_bootstrap.cc:492] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:54.505182 30633 ts_tablet_manager.cc:1367] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.003s
I20240502 08:17:54.505260 30749 tablet_bootstrap.cc:492] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:54.506680 30633 raft_consensus.cc:348] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:17:54.506843 30749 tablet_bootstrap.cc:654] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.506978 30633 raft_consensus.cc:374] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.507139 30633 raft_consensus.cc:729] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.507459 30633 consensus_queue.cc:260] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:17:54.508813 30633 ts_tablet_manager.cc:1398] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:54.509335 30628 tablet_bootstrap.cc:492] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:17:54.510615 30748 tablet_bootstrap.cc:492] T 168e71e71b0340fd953d5644b8b8c028 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:54.510677 27110 catalog_manager.cc:5576] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309) does not have the latest schema for tablet 9ffcfdb358274bb680fbb8afae05691a (table rebalancer_dynamic_table [id=9bc7251a63034f0bb05e56e417d7dfa8]). Expected version 1 got 0
I20240502 08:17:54.511015 30628 tablet_bootstrap.cc:654] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.512187 30748 tablet_bootstrap.cc:654] T 168e71e71b0340fd953d5644b8b8c028 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.519563 30751 tablet_bootstrap.cc:492] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:54.519744 30751 ts_tablet_manager.cc:1367] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240502 08:17:54.521373 30751 raft_consensus.cc:348] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:54.521638 30751 raft_consensus.cc:374] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.521764 30751 raft_consensus.cc:729] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.522039 30751 consensus_queue.cc:260] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:54.522616 30751 ts_tablet_manager.cc:1398] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:54.523703 30753 tablet_bootstrap.cc:492] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:54.525673 30753 tablet_bootstrap.cc:654] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.526575 30628 tablet_bootstrap.cc:492] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:17:54.526793 30628 ts_tablet_manager.cc:1367] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.009s
I20240502 08:17:54.527771 30748 tablet_bootstrap.cc:492] T 168e71e71b0340fd953d5644b8b8c028 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:17:54.527684 30756 tablet.cc:1722] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:54.527915 30748 ts_tablet_manager.cc:1367] T 168e71e71b0340fd953d5644b8b8c028 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240502 08:17:54.527719 30628 raft_consensus.cc:348] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:54.528033 30628 raft_consensus.cc:374] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.528100 30628 raft_consensus.cc:729] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.528357 30628 consensus_queue.cc:260] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:54.528944 30756 tablet.cc:1620] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442: MemRowSet was empty: no flush needed.
I20240502 08:17:54.529122 30748 raft_consensus.cc:348] T 168e71e71b0340fd953d5644b8b8c028 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:54.529378 30748 raft_consensus.cc:374] T 168e71e71b0340fd953d5644b8b8c028 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.529516 30748 raft_consensus.cc:729] T 168e71e71b0340fd953d5644b8b8c028 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.529819 30748 consensus_queue.cc:260] T 168e71e71b0340fd953d5644b8b8c028 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:54.531126 30748 ts_tablet_manager.cc:1398] T 168e71e71b0340fd953d5644b8b8c028 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:54.531419 30748 tablet_bootstrap.cc:492] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:54.533100 30748 tablet_bootstrap.cc:654] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.534255 27110 catalog_manager.cc:5576] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309) does not have the latest schema for tablet 9ffcfdb358274bb680fbb8afae05691a (table rebalancer_dynamic_table [id=9bc7251a63034f0bb05e56e417d7dfa8]). Expected version 1 got 0
I20240502 08:17:54.534665 30628 ts_tablet_manager.cc:1398] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 08:17:54.535704 30628 tablet_bootstrap.cc:492] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:17:54.540984 30753 tablet_bootstrap.cc:492] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:54.541186 30753 ts_tablet_manager.cc:1367] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.004s
I20240502 08:17:54.542064 30753 raft_consensus.cc:348] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:54.542325 30753 raft_consensus.cc:374] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.542462 30753 raft_consensus.cc:729] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.542701 30753 consensus_queue.cc:260] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:54.543802 30628 tablet_bootstrap.cc:654] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.544417 30753 ts_tablet_manager.cc:1398] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:54.545650 30628 tablet_bootstrap.cc:492] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:17:54.545768 30628 ts_tablet_manager.cc:1367] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20240502 08:17:54.546618 30628 raft_consensus.cc:348] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:17:54.546787 30628 raft_consensus.cc:374] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.546875 30628 raft_consensus.cc:729] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.547112 30628 consensus_queue.cc:260] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:17:54.550221 30628 ts_tablet_manager.cc:1398] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:54.552325 30749 tablet_bootstrap.cc:492] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:54.552589 30749 ts_tablet_manager.cc:1367] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.000s
I20240502 08:17:54.553580 30749 raft_consensus.cc:348] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:17:54.553859 30749 raft_consensus.cc:374] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.553951 30749 raft_consensus.cc:729] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.554304 30749 consensus_queue.cc:260] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:17:54.555012 30749 ts_tablet_manager.cc:1398] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:54.556968 30748 tablet_bootstrap.cc:492] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:17:54.557160 30748 ts_tablet_manager.cc:1367] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.006s
I20240502 08:17:54.558173 30748 raft_consensus.cc:348] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:54.558401 30748 raft_consensus.cc:374] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.558478 30748 raft_consensus.cc:729] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.558727 30748 consensus_queue.cc:260] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:54.559353 30748 ts_tablet_manager.cc:1398] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:54.591737 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:17:54.600371 30746 tablet.cc:1722] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:54.600821 30746 tablet.cc:1620] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:17:54.604187 30747 tablet.cc:1722] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:54.604642 30747 tablet.cc:1620] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:17:54.606676 30756 tablet.cc:1722] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:54.607108 30756 tablet.cc:1620] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442: MemRowSet was empty: no flush needed.
I20240502 08:17:54.613512 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=9bc7251a63034f0bb05e56e417d7dfa8] alter complete (version 2)
I20240502 08:17:54.634254 27116 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 08:17:54.634434 27116 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 08:17:54.637674 27116 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet f11f7ab800ba4466acb6cf9fefd19c52
I20240502 08:17:54.638255 27116 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet fbec19764eb84f1ca920a785d8b56d8b
I20240502 08:17:54.638533 27236 tablet_service.cc:1514] Processing DeleteTablet for tablet f11f7ab800ba4466acb6cf9fefd19c52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:54.638721 27499 tablet_service.cc:1514] Processing DeleteTablet for tablet f11f7ab800ba4466acb6cf9fefd19c52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:17:54.638846 30524 tablet_replica.cc:330] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:54.638989 30521 tablet_replica.cc:330] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:17:54.639067 30524 raft_consensus.cc:2229] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:54.639072 27630 tablet_service.cc:1514] Processing DeleteTablet for tablet fbec19764eb84f1ca920a785d8b56d8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:54.639181 30521 raft_consensus.cc:2229] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:54.639401 30524 raft_consensus.cc:2258] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.639439 30521 raft_consensus.cc:2258] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.640058 30521 ts_tablet_manager.cc:1875] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.640151 30524 ts_tablet_manager.cc:1875] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.642218 30522 tablet_replica.cc:330] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:54.642421 30522 raft_consensus.cc:2229] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:54.642760 30522 raft_consensus.cc:2258] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.643395 30522 ts_tablet_manager.cc:1875] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.645247 30521 ts_tablet_manager.cc:1888] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:54.645354 30521 log.cc:1196] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/f11f7ab800ba4466acb6cf9fefd19c52
I20240502 08:17:54.646066 27116 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 14446aeb355e4b68bcc81da74d22ad9b
I20240502 08:17:54.646126 30521 ts_tablet_manager.cc:1909] T f11f7ab800ba4466acb6cf9fefd19c52 P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:17:54.646135 27236 tablet_service.cc:1514] Processing DeleteTablet for tablet fbec19764eb84f1ca920a785d8b56d8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:54.646436 27370 tablet_service.cc:1514] Processing DeleteTablet for tablet f11f7ab800ba4466acb6cf9fefd19c52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:54.646723 30523 tablet_replica.cc:330] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:54.646751 27368 tablet_service.cc:1514] Processing DeleteTablet for tablet fbec19764eb84f1ca920a785d8b56d8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:54.646939 30523 raft_consensus.cc:2229] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:54.647183 30523 raft_consensus.cc:2258] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.647873 30523 ts_tablet_manager.cc:1875] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.648165 30524 ts_tablet_manager.cc:1888] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:54.648154 27630 tablet_service.cc:1514] Processing DeleteTablet for tablet 14446aeb355e4b68bcc81da74d22ad9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:54.648368 30524 log.cc:1196] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/f11f7ab800ba4466acb6cf9fefd19c52
I20240502 08:17:54.648434 27236 tablet_service.cc:1514] Processing DeleteTablet for tablet 14446aeb355e4b68bcc81da74d22ad9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:54.648800 30524 ts_tablet_manager.cc:1909] T f11f7ab800ba4466acb6cf9fefd19c52 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:54.649556 30524 tablet_replica.cc:330] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:54.649793 30524 raft_consensus.cc:2229] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:54.650032 30524 raft_consensus.cc:2258] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.650631 30524 ts_tablet_manager.cc:1875] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.650998 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet f11f7ab800ba4466acb6cf9fefd19c52 (table rebalance_test_table_8 [id=9eb13e13b76544e79bcde9a39e3b2cd4]) successfully deleted
I20240502 08:17:54.651048 27499 tablet_service.cc:1514] Processing DeleteTablet for tablet 14446aeb355e4b68bcc81da74d22ad9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:17:54.656486 30524 ts_tablet_manager.cc:1888] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:54.656612 30524 log.cc:1196] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/fbec19764eb84f1ca920a785d8b56d8b
I20240502 08:17:54.656999 30524 ts_tablet_manager.cc:1909] T fbec19764eb84f1ca920a785d8b56d8b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:54.657426 30522 ts_tablet_manager.cc:1888] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:54.657598 30522 log.cc:1196] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/fbec19764eb84f1ca920a785d8b56d8b
I20240502 08:17:54.657886 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet fbec19764eb84f1ca920a785d8b56d8b (table rebalance_test_table_8 [id=9eb13e13b76544e79bcde9a39e3b2cd4]) successfully deleted
I20240502 08:17:54.658156 30522 ts_tablet_manager.cc:1909] T fbec19764eb84f1ca920a785d8b56d8b P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:54.658236 30524 tablet_replica.cc:330] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:54.658421 30524 raft_consensus.cc:2229] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:54.658674 30524 raft_consensus.cc:2258] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.659271 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet fbec19764eb84f1ca920a785d8b56d8b (table rebalance_test_table_8 [id=9eb13e13b76544e79bcde9a39e3b2cd4]) successfully deleted
I20240502 08:17:54.659373 30524 ts_tablet_manager.cc:1875] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.659834 30522 tablet_replica.cc:330] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:54.660125 30522 raft_consensus.cc:2229] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:54.660544 30522 raft_consensus.cc:2258] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.660579 30523 ts_tablet_manager.cc:1888] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:54.661831 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet f11f7ab800ba4466acb6cf9fefd19c52 (table rebalance_test_table_8 [id=9eb13e13b76544e79bcde9a39e3b2cd4]) successfully deleted
I20240502 08:17:54.662127 30522 ts_tablet_manager.cc:1875] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.662266 30523 log.cc:1196] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/f11f7ab800ba4466acb6cf9fefd19c52
I20240502 08:17:54.662676 30523 ts_tablet_manager.cc:1909] T f11f7ab800ba4466acb6cf9fefd19c52 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:54.661728 30521 tablet_replica.cc:330] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:17:54.663767 30521 raft_consensus.cc:2229] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:54.663980 30521 raft_consensus.cc:2258] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.665603 30523 tablet_replica.cc:330] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:54.665694 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet f11f7ab800ba4466acb6cf9fefd19c52 (table rebalance_test_table_8 [id=9eb13e13b76544e79bcde9a39e3b2cd4]) successfully deleted
I20240502 08:17:54.665822 30523 raft_consensus.cc:2229] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:54.666386 30521 ts_tablet_manager.cc:1875] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.666653 30524 ts_tablet_manager.cc:1888] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:54.666764 30524 log.cc:1196] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/14446aeb355e4b68bcc81da74d22ad9b
I20240502 08:17:54.667207 30524 ts_tablet_manager.cc:1909] T 14446aeb355e4b68bcc81da74d22ad9b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:54.667984 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet 14446aeb355e4b68bcc81da74d22ad9b (table rebalance_test_table_8 [id=9eb13e13b76544e79bcde9a39e3b2cd4]) successfully deleted
I20240502 08:17:54.669451 30522 ts_tablet_manager.cc:1888] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:54.669581 30522 log.cc:1196] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/14446aeb355e4b68bcc81da74d22ad9b
I20240502 08:17:54.669953 30522 ts_tablet_manager.cc:1909] T 14446aeb355e4b68bcc81da74d22ad9b P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:54.670845 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 14446aeb355e4b68bcc81da74d22ad9b (table rebalance_test_table_8 [id=9eb13e13b76544e79bcde9a39e3b2cd4]) successfully deleted
I20240502 08:17:54.670876 30523 raft_consensus.cc:2258] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.671527 30523 ts_tablet_manager.cc:1875] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.672092 30521 ts_tablet_manager.cc:1888] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:54.672202 30521 log.cc:1196] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/14446aeb355e4b68bcc81da74d22ad9b
I20240502 08:17:54.672536 30521 ts_tablet_manager.cc:1909] T 14446aeb355e4b68bcc81da74d22ad9b P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:17:54.673241 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 14446aeb355e4b68bcc81da74d22ad9b (table rebalance_test_table_8 [id=9eb13e13b76544e79bcde9a39e3b2cd4]) successfully deleted
W20240502 08:17:54.674551 27436 tablet.cc:2369] T 168e71e71b0340fd953d5644b8b8c028 P 83e171a5e16c4008923f1d43fc5ce7b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:54.677102 30523 ts_tablet_manager.cc:1888] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:54.677209 30523 log.cc:1196] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/fbec19764eb84f1ca920a785d8b56d8b
I20240502 08:17:54.677547 30523 ts_tablet_manager.cc:1909] T fbec19764eb84f1ca920a785d8b56d8b P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:54.679653 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet fbec19764eb84f1ca920a785d8b56d8b (table rebalance_test_table_8 [id=9eb13e13b76544e79bcde9a39e3b2cd4]) successfully deleted
I20240502 08:17:54.707618 30546 raft_consensus.cc:482] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:54.707792 30546 raft_consensus.cc:504] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:54.708745 30546 leader_election.cc:288] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:54.709465 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67c99e57a73443da13527ed33326a4e" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:17:54.709683 30290 raft_consensus.cc:2454] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:54.710108 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67c99e57a73443da13527ed33326a4e" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:17:54.710229 27191 leader_election.cc:302] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:17:54.710342 30160 raft_consensus.cc:2454] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:17:54.710909 29863 raft_consensus.cc:2781] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:54.710994 29863 raft_consensus.cc:482] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:54.711095 29863 raft_consensus.cc:3037] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.713204 29863 raft_consensus.cc:504] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:54.716885 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67c99e57a73443da13527ed33326a4e" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:17:54.717144 30160 raft_consensus.cc:3037] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.717837 27116 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:17:54.718358 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67c99e57a73443da13527ed33326a4e" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:17:54.718636 30290 raft_consensus.cc:3037] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.730957 30746 tablet.cc:1722] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:54.735394 30756 tablet.cc:1722] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:54.735761 30552 raft_consensus.cc:482] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:54.735904 30552 raft_consensus.cc:504] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:17:54.736171 30747 tablet.cc:1722] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:54.736757 30552 leader_election.cc:288] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:17:54.740711 29863 leader_election.cc:288] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:54.741382 30290 raft_consensus.cc:2454] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:54.741840 27191 leader_election.cc:302] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:17:54.742218 29863 raft_consensus.cc:2781] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:54.742960 29863 raft_consensus.cc:686] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:17:54.743389 29863 consensus_queue.cc:237] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:54.743893 30160 raft_consensus.cc:2454] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:17:54.747336 30732 raft_consensus.cc:482] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:54.747479 30732 raft_consensus.cc:504] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:54.748484 30732 leader_election.cc:288] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:54.753542 27113 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=9bc7251a63034f0bb05e56e417d7dfa8] alter complete (version 3)
I20240502 08:17:54.753830 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a26e01d42b429694641f0ceca11265" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:17:54.754144 30028 raft_consensus.cc:2454] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:54.755156 27323 leader_election.cc:302] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:54.755498 30552 raft_consensus.cc:2781] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:54.755606 30552 raft_consensus.cc:482] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:54.755681 30552 raft_consensus.cc:3037] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.760406 30552 raft_consensus.cc:504] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:17:54.761358 30552 leader_election.cc:288] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:17:54.762089 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a26e01d42b429694641f0ceca11265" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:17:54.762324 30028 raft_consensus.cc:3037] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.765146 30028 raft_consensus.cc:2454] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:54.765666 27323 leader_election.cc:302] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:54.765977 30552 raft_consensus.cc:2781] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:54.766259 30552 raft_consensus.cc:686] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:17:54.766566 30552 consensus_queue.cc:237] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:17:54.780293 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a26e01d42b429694641f0ceca11265" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:17:54.780530 30421 raft_consensus.cc:3037] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.783195 30421 raft_consensus.cc:2454] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:54.784031 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a26e01d42b429694641f0ceca11265" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:17:54.784232 30422 raft_consensus.cc:2362] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1. Re-sending same reply.
I20240502 08:17:54.786463 27112 catalog_manager.cc:5484] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:54.787739 27114 catalog_manager.cc:5484] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:54.792721 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "168e71e71b0340fd953d5644b8b8c028" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:17:54.793049 30160 raft_consensus.cc:2454] T 168e71e71b0340fd953d5644b8b8c028 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:17:54.793576 30358 leader_election.cc:302] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:54.793900 30627 raft_consensus.cc:2781] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:54.794067 30627 raft_consensus.cc:482] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:54.794170 30627 raft_consensus.cc:3037] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.796913 30627 raft_consensus.cc:504] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:54.798691 30627 leader_election.cc:288] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:54.799293 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "168e71e71b0340fd953d5644b8b8c028" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:17:54.799546 30160 raft_consensus.cc:3037] T 168e71e71b0340fd953d5644b8b8c028 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.802238 30160 raft_consensus.cc:2454] T 168e71e71b0340fd953d5644b8b8c028 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:17:54.802803 30358 leader_election.cc:302] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:54.804010 30627 raft_consensus.cc:2781] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:54.804603 30627 raft_consensus.cc:686] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:17:54.804936 30627 consensus_queue.cc:237] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:54.805795 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "168e71e71b0340fd953d5644b8b8c028" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:54.806257 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "168e71e71b0340fd953d5644b8b8c028" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:54.806699 27390 raft_consensus.cc:3037] T 168e71e71b0340fd953d5644b8b8c028 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.809445 27390 raft_consensus.cc:2454] T 168e71e71b0340fd953d5644b8b8c028 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:17:54.818944 27107 catalog_manager.cc:5484] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:54.874202 27116 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:17:54.874480 27116 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:17:54.880945 27116 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 9ffcfdb358274bb680fbb8afae05691a
I20240502 08:17:54.882045 30007 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ffcfdb358274bb680fbb8afae05691a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:58374
I20240502 08:17:54.885530 30401 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ffcfdb358274bb680fbb8afae05691a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:56694
I20240502 08:17:54.886391 27630 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ffcfdb358274bb680fbb8afae05691a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:54 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:54.886705 30522 tablet_replica.cc:330] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:54.886919 30522 raft_consensus.cc:2229] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:54.887184 30522 raft_consensus.cc:2258] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.888911 30522 ts_tablet_manager.cc:1875] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.893303 30806 tablet_replica.cc:330] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:17:54.895462 30806 raft_consensus.cc:2229] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:54.895798 30806 raft_consensus.cc:2258] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.896643 30806 ts_tablet_manager.cc:1875] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.906924 30522 ts_tablet_manager.cc:1888] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:54.907161 30522 log.cc:1196] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/9ffcfdb358274bb680fbb8afae05691a
I20240502 08:17:54.908118 30522 ts_tablet_manager.cc:1909] T 9ffcfdb358274bb680fbb8afae05691a P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:54.908414 30806 ts_tablet_manager.cc:1888] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:54.908566 30806 log.cc:1196] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/9ffcfdb358274bb680fbb8afae05691a
I20240502 08:17:54.909180 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 9ffcfdb358274bb680fbb8afae05691a (table rebalancer_dynamic_table_new_name [id=9bc7251a63034f0bb05e56e417d7dfa8]) successfully deleted
I20240502 08:17:54.909626 30806 ts_tablet_manager.cc:1909] T 9ffcfdb358274bb680fbb8afae05691a P 1f0bf4bc3c5146539146b825b048bff3: Deleting consensus metadata
I20240502 08:17:54.910852 30807 tablet_replica.cc:330] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:17:54.910991 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet 9ffcfdb358274bb680fbb8afae05691a (table rebalancer_dynamic_table_new_name [id=9bc7251a63034f0bb05e56e417d7dfa8]) successfully deleted
I20240502 08:17:54.911440 30807 raft_consensus.cc:2229] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:54.912410 30807 raft_consensus.cc:2258] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:54.914307 30807 ts_tablet_manager.cc:1875] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:54.935171 30807 ts_tablet_manager.cc:1888] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:54.936975 30807 log.cc:1196] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/9ffcfdb358274bb680fbb8afae05691a
I20240502 08:17:54.937747 30807 ts_tablet_manager.cc:1909] T 9ffcfdb358274bb680fbb8afae05691a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting consensus metadata
I20240502 08:17:54.940677 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet 9ffcfdb358274bb680fbb8afae05691a (table rebalancer_dynamic_table_new_name [id=9bc7251a63034f0bb05e56e417d7dfa8]) successfully deleted
I20240502 08:17:54.967314 30401 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:54.969528 27236 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:17:54.996443 27368 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:17:55.003878 30140 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:55.005424 27116 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:17:55.015003 30007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:55.022651 27368 tablet_service.cc:1467] Processing CreateTablet for tablet b2aa620b1bd747a79ec84d2cf7b727ef (DEFAULT_TABLE table=rebalancer_dynamic_table [id=046e5e3d46974da29f94c4f9d2803571]), partition=
I20240502 08:17:55.022953 30401 tablet_service.cc:1467] Processing CreateTablet for tablet b2aa620b1bd747a79ec84d2cf7b727ef (DEFAULT_TABLE table=rebalancer_dynamic_table [id=046e5e3d46974da29f94c4f9d2803571]), partition=
I20240502 08:17:55.022998 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2aa620b1bd747a79ec84d2cf7b727ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.023341 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2aa620b1bd747a79ec84d2cf7b727ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.028165 30007 tablet_service.cc:1467] Processing CreateTablet for tablet b2aa620b1bd747a79ec84d2cf7b727ef (DEFAULT_TABLE table=rebalancer_dynamic_table [id=046e5e3d46974da29f94c4f9d2803571]), partition=
I20240502 08:17:55.028542 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2aa620b1bd747a79ec84d2cf7b727ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.032764 30749 tablet_bootstrap.cc:492] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:55.034555 30749 tablet_bootstrap.cc:654] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.050706 30749 tablet_bootstrap.cc:492] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:55.051328 30749 ts_tablet_manager.cc:1367] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240502 08:17:55.053273 30749 raft_consensus.cc:348] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:55.053663 30749 raft_consensus.cc:374] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:55.053807 30749 raft_consensus.cc:729] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:55.055935 30749 consensus_queue.cc:260] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:55.064952 30268 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:55.087072 30749 ts_tablet_manager.cc:1398] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20240502 08:17:55.104234 27499 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:17:55.105111 30628 tablet_bootstrap.cc:492] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:17:55.107115 30628 tablet_bootstrap.cc:654] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.124833 30628 tablet_bootstrap.cc:492] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:17:55.125111 30628 ts_tablet_manager.cc:1367] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.000s
I20240502 08:17:55.126180 30628 raft_consensus.cc:348] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:55.126655 30762 raft_consensus.cc:482] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:55.126827 30762 raft_consensus.cc:504] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:55.127802 30762 leader_election.cc:288] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:17:55.129145 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2aa620b1bd747a79ec84d2cf7b727ef" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:17:55.131177 30628 raft_consensus.cc:374] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:55.131374 30628 raft_consensus.cc:729] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:17:55.131744 30628 consensus_queue.cc:260] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:55.132290 30422 raft_consensus.cc:2454] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:55.133191 27323 leader_election.cc:302] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:55.133596 30762 raft_consensus.cc:2781] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:55.133718 30762 raft_consensus.cc:482] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:55.133803 30762 raft_consensus.cc:3037] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:55.137132 30762 raft_consensus.cc:504] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:55.143903 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2aa620b1bd747a79ec84d2cf7b727ef" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:17:55.144098 30422 raft_consensus.cc:3037] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:55.148077 30422 raft_consensus.cc:2454] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:55.148578 27323 leader_election.cc:302] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:55.148913 30552 raft_consensus.cc:2781] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:55.149852 30762 leader_election.cc:288] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:17:55.150525 30628 ts_tablet_manager.cc:1398] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240502 08:17:55.151388 27116 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 08:17:55.151612 27116 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 08:17:55.152204 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2aa620b1bd747a79ec84d2cf7b727ef" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:17:55.156069 27116 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet fddf0ae2372b492ba67f8a4d7de47129
I20240502 08:17:55.156687 27116 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 0bf33bf3e4d542f287ba670c2348b34d
I20240502 08:17:55.157174 27116 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 7e8228b23331486592b7e958a6c52f4c
I20240502 08:17:55.158454 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2aa620b1bd747a79ec84d2cf7b727ef" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:17:55.159305 27630 tablet_service.cc:1514] Processing DeleteTablet for tablet fddf0ae2372b492ba67f8a4d7de47129 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:55.159417 27499 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf33bf3e4d542f287ba670c2348b34d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:17:55.159600 30522 tablet_replica.cc:330] T fddf0ae2372b492ba67f8a4d7de47129 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:55.159690 27629 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf33bf3e4d542f287ba670c2348b34d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:55.159832 30522 raft_consensus.cc:2229] T fddf0ae2372b492ba67f8a4d7de47129 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:55.159911 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e8228b23331486592b7e958a6c52f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:55.160075 30522 raft_consensus.cc:2258] T fddf0ae2372b492ba67f8a4d7de47129 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.160108 30521 tablet_replica.cc:330] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:17:55.160187 27236 tablet_service.cc:1514] Processing DeleteTablet for tablet fddf0ae2372b492ba67f8a4d7de47129 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:55.160293 30521 raft_consensus.cc:2229] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:55.160499 30524 tablet_replica.cc:330] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:55.160516 30521 raft_consensus.cc:2258] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.160779 30522 ts_tablet_manager.cc:1875] T fddf0ae2372b492ba67f8a4d7de47129 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.160789 30524 raft_consensus.cc:2229] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:55.161095 30524 raft_consensus.cc:2258] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.161175 30521 ts_tablet_manager.cc:1875] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.161939 30524 ts_tablet_manager.cc:1875] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.162153 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf33bf3e4d542f287ba670c2348b34d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:55.165083 27237 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e8228b23331486592b7e958a6c52f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:55.166044 27368 tablet_service.cc:1514] Processing DeleteTablet for tablet fddf0ae2372b492ba67f8a4d7de47129 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:55.166182 27370 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e8228b23331486592b7e958a6c52f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:55.166383 30523 tablet_replica.cc:330] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
W20240502 08:17:55.166570 27323 leader_election.cc:341] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:55.166646 30523 raft_consensus.cc:2229] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Raft consensus shutting down.
W20240502 08:17:55.166898 27323 leader_election.cc:341] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:55.166937 30523 raft_consensus.cc:2258] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.167191 30522 ts_tablet_manager.cc:1888] T fddf0ae2372b492ba67f8a4d7de47129 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:55.167320 30522 log.cc:1196] T fddf0ae2372b492ba67f8a4d7de47129 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/fddf0ae2372b492ba67f8a4d7de47129
I20240502 08:17:55.167697 30522 ts_tablet_manager.cc:1909] T fddf0ae2372b492ba67f8a4d7de47129 P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:55.167749 30523 ts_tablet_manager.cc:1875] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.168648 30522 tablet_replica.cc:330] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:55.168833 30522 raft_consensus.cc:2229] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:55.169029 30522 raft_consensus.cc:2258] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.169607 30522 ts_tablet_manager.cc:1875] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.170131 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet fddf0ae2372b492ba67f8a4d7de47129 (table rebalance_test_table_10 [id=c9beaea1ee3d4d3b93cac242b3ff8722]) successfully deleted
I20240502 08:17:55.174120 30521 ts_tablet_manager.cc:1888] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:55.174258 30521 log.cc:1196] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/0bf33bf3e4d542f287ba670c2348b34d
I20240502 08:17:55.174690 30521 ts_tablet_manager.cc:1909] T 0bf33bf3e4d542f287ba670c2348b34d P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:17:55.175652 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 0bf33bf3e4d542f287ba670c2348b34d (table rebalance_test_table_10 [id=c9beaea1ee3d4d3b93cac242b3ff8722]) successfully deleted
I20240502 08:17:55.175949 30823 tablet_bootstrap.cc:492] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:55.177552 30823 tablet_bootstrap.cc:654] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.185202 30522 ts_tablet_manager.cc:1888] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:55.185340 30522 log.cc:1196] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/0bf33bf3e4d542f287ba670c2348b34d
I20240502 08:17:55.185827 30522 ts_tablet_manager.cc:1909] T 0bf33bf3e4d542f287ba670c2348b34d P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:55.186972 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 0bf33bf3e4d542f287ba670c2348b34d (table rebalance_test_table_10 [id=c9beaea1ee3d4d3b93cac242b3ff8722]) successfully deleted
I20240502 08:17:55.188164 30523 ts_tablet_manager.cc:1888] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:55.188691 30524 ts_tablet_manager.cc:1888] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:55.188927 30523 log.cc:1196] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/fddf0ae2372b492ba67f8a4d7de47129
I20240502 08:17:55.190505 30524 log.cc:1196] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/fddf0ae2372b492ba67f8a4d7de47129
I20240502 08:17:55.190835 30523 ts_tablet_manager.cc:1909] T fddf0ae2372b492ba67f8a4d7de47129 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:55.191702 30524 ts_tablet_manager.cc:1909] T fddf0ae2372b492ba67f8a4d7de47129 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:55.192860 30522 tablet_replica.cc:330] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:55.192838 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet fddf0ae2372b492ba67f8a4d7de47129 (table rebalance_test_table_10 [id=c9beaea1ee3d4d3b93cac242b3ff8722]) successfully deleted
I20240502 08:17:55.193069 30522 raft_consensus.cc:2229] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:55.193300 30522 raft_consensus.cc:2258] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.194177 30522 ts_tablet_manager.cc:1875] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.195170 30523 tablet_replica.cc:330] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:55.195595 30523 raft_consensus.cc:2229] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:55.196005 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet fddf0ae2372b492ba67f8a4d7de47129 (table rebalance_test_table_10 [id=c9beaea1ee3d4d3b93cac242b3ff8722]) successfully deleted
I20240502 08:17:55.196378 30524 tablet_replica.cc:330] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:55.198437 30523 raft_consensus.cc:2258] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.198755 30524 raft_consensus.cc:2229] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:55.203681 30523 ts_tablet_manager.cc:1875] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.204047 30524 raft_consensus.cc:2258] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.204694 30522 ts_tablet_manager.cc:1888] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:55.204833 30522 log.cc:1196] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/7e8228b23331486592b7e958a6c52f4c
I20240502 08:17:55.205089 30524 ts_tablet_manager.cc:1875] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.205695 30522 ts_tablet_manager.cc:1909] T 7e8228b23331486592b7e958a6c52f4c P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:55.206701 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 7e8228b23331486592b7e958a6c52f4c (table rebalance_test_table_10 [id=c9beaea1ee3d4d3b93cac242b3ff8722]) successfully deleted
I20240502 08:17:55.208132 30762 consensus_queue.cc:1035] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:55.209853 29863 consensus_queue.cc:1035] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:55.212590 29863 consensus_queue.cc:1035] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:55.231051 30523 ts_tablet_manager.cc:1888] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:55.231484 30524 ts_tablet_manager.cc:1888] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:55.231684 30524 log.cc:1196] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/0bf33bf3e4d542f287ba670c2348b34d
I20240502 08:17:55.232240 30524 ts_tablet_manager.cc:1909] T 0bf33bf3e4d542f287ba670c2348b34d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:55.232401 30523 log.cc:1196] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/7e8228b23331486592b7e958a6c52f4c
I20240502 08:17:55.232914 30552 raft_consensus.cc:686] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:17:55.232915 30523 ts_tablet_manager.cc:1909] T 7e8228b23331486592b7e958a6c52f4c P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:55.234138 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet 0bf33bf3e4d542f287ba670c2348b34d (table rebalance_test_table_10 [id=c9beaea1ee3d4d3b93cac242b3ff8722]) successfully deleted
I20240502 08:17:55.233232 30552 consensus_queue.cc:237] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:55.235515 30524 tablet_replica.cc:330] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:55.235675 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 7e8228b23331486592b7e958a6c52f4c (table rebalance_test_table_10 [id=c9beaea1ee3d4d3b93cac242b3ff8722]) successfully deleted
I20240502 08:17:55.236081 30524 raft_consensus.cc:2229] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:55.236407 30524 raft_consensus.cc:2258] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.237162 30524 ts_tablet_manager.cc:1875] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.256090 30823 tablet_bootstrap.cc:492] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:55.256269 30823 ts_tablet_manager.cc:1367] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.080s user 0.026s sys 0.010s
I20240502 08:17:55.257174 30823 raft_consensus.cc:348] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:55.257447 30823 raft_consensus.cc:374] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:55.257539 30823 raft_consensus.cc:729] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:55.257803 30823 consensus_queue.cc:260] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:55.258395 30823 ts_tablet_manager.cc:1398] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:17:55.268240 30524 ts_tablet_manager.cc:1888] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:55.270117 30524 log.cc:1196] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/7e8228b23331486592b7e958a6c52f4c
I20240502 08:17:55.270859 30524 ts_tablet_manager.cc:1909] T 7e8228b23331486592b7e958a6c52f4c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:55.272609 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet 7e8228b23331486592b7e958a6c52f4c (table rebalance_test_table_10 [id=c9beaea1ee3d4d3b93cac242b3ff8722]) successfully deleted
I20240502 08:17:55.275336 30762 consensus_queue.cc:1035] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:55.298131 27113 catalog_manager.cc:5484] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:55.308369 27631 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:55.400045 27113 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:55.405997 30829 raft_consensus.cc:482] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:55.406183 30829 raft_consensus.cc:504] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:17:55.406816 30829 leader_election.cc:288] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:55.407581 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2aa620b1bd747a79ec84d2cf7b727ef" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:55.408335 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2aa620b1bd747a79ec84d2cf7b727ef" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:17:55.408537 30422 raft_consensus.cc:2379] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f0bf4bc3c5146539146b825b048bff3 in current term 1: Already voted for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in this term.
I20240502 08:17:55.410269 29961 leader_election.cc:302] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:17:55.411561 30627 consensus_queue.cc:1035] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:55.412065 30829 raft_consensus.cc:3037] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:55.416289 30007 tablet_service.cc:1467] Processing CreateTablet for tablet f63eec7e63fe4e71b1958903e0d2cb00 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=e57c96062aca4609bc2d923cb7a30a29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:55.416887 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63eec7e63fe4e71b1958903e0d2cb00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.417333 27113 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:17:55.421149 27370 tablet_service.cc:1467] Processing CreateTablet for tablet f63eec7e63fe4e71b1958903e0d2cb00 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=e57c96062aca4609bc2d923cb7a30a29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:55.421610 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63eec7e63fe4e71b1958903e0d2cb00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.423162 30140 tablet_service.cc:1467] Processing CreateTablet for tablet 83993ff2073842988937597dd3999d0d (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=e57c96062aca4609bc2d923cb7a30a29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:55.423590 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83993ff2073842988937597dd3999d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.428093 30749 tablet_bootstrap.cc:492] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:55.428581 30268 tablet_service.cc:1467] Processing CreateTablet for tablet f63eec7e63fe4e71b1958903e0d2cb00 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=e57c96062aca4609bc2d923cb7a30a29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:55.429167 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63eec7e63fe4e71b1958903e0d2cb00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.429834 30749 tablet_bootstrap.cc:654] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.435086 30008 tablet_service.cc:1467] Processing CreateTablet for tablet 83993ff2073842988937597dd3999d0d (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=e57c96062aca4609bc2d923cb7a30a29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:55.435222 30749 tablet_bootstrap.cc:492] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:55.435560 30008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83993ff2073842988937597dd3999d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.435626 30749 ts_tablet_manager.cc:1367] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20240502 08:17:55.436297 30749 raft_consensus.cc:348] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:55.436493 30749 raft_consensus.cc:374] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:55.436547 30749 raft_consensus.cc:729] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:55.436743 30749 consensus_queue.cc:260] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:55.437179 30749 ts_tablet_manager.cc:1398] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:55.438999 30422 raft_consensus.cc:1261] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:55.439879 30826 consensus_queue.cc:1035] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:55.441892 30627 consensus_queue.cc:1035] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:55.452988 30829 raft_consensus.cc:2726] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:55.454525 30027 raft_consensus.cc:1261] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:55.455602 30552 consensus_queue.cc:1035] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:55.465174 30401 tablet_service.cc:1467] Processing CreateTablet for tablet 83993ff2073842988937597dd3999d0d (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=e57c96062aca4609bc2d923cb7a30a29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:55.465642 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83993ff2073842988937597dd3999d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.465963 30006 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb679bf525a4a2ebafeb73740230569 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=e57c96062aca4609bc2d923cb7a30a29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:55.466405 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb679bf525a4a2ebafeb73740230569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.471567 30628 tablet_bootstrap.cc:492] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:17:55.473071 30628 tablet_bootstrap.cc:654] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.474355 30139 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb679bf525a4a2ebafeb73740230569 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=e57c96062aca4609bc2d923cb7a30a29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:55.474442 30402 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb679bf525a4a2ebafeb73740230569 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=e57c96062aca4609bc2d923cb7a30a29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:55.474779 30139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb679bf525a4a2ebafeb73740230569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.474804 30402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb679bf525a4a2ebafeb73740230569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.479290 30823 tablet_bootstrap.cc:492] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:55.480859 30823 tablet_bootstrap.cc:654] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.488235 30851 tablet_bootstrap.cc:492] T 83993ff2073842988937597dd3999d0d P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:55.490698 30851 tablet_bootstrap.cc:654] T 83993ff2073842988937597dd3999d0d P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.491729 30852 tablet_bootstrap.cc:492] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:55.495250 30849 tablet.cc:1722] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:55.496177 30849 tablet.cc:1620] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:17:55.497920 30852 tablet_bootstrap.cc:654] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.500092 30851 tablet_bootstrap.cc:492] T 83993ff2073842988937597dd3999d0d P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:17:55.500296 30851 ts_tablet_manager.cc:1367] T 83993ff2073842988937597dd3999d0d P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 08:17:55.500373 30628 tablet_bootstrap.cc:492] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:17:55.500531 30628 ts_tablet_manager.cc:1367] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.002s
I20240502 08:17:55.501230 30851 raft_consensus.cc:348] T 83993ff2073842988937597dd3999d0d P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:55.501303 30628 raft_consensus.cc:348] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:55.501484 30628 raft_consensus.cc:374] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:55.501580 30628 raft_consensus.cc:729] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:17:55.501586 30851 raft_consensus.cc:374] T 83993ff2073842988937597dd3999d0d P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:55.501732 30851 raft_consensus.cc:729] T 83993ff2073842988937597dd3999d0d P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:17:55.501837 30628 consensus_queue.cc:260] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:55.501987 30851 consensus_queue.cc:260] T 83993ff2073842988937597dd3999d0d P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:55.502475 30628 ts_tablet_manager.cc:1398] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:55.502724 30628 tablet_bootstrap.cc:492] T 8bb679bf525a4a2ebafeb73740230569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:17:55.502768 30851 ts_tablet_manager.cc:1398] T 83993ff2073842988937597dd3999d0d P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:55.504038 30628 tablet_bootstrap.cc:654] T 8bb679bf525a4a2ebafeb73740230569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.505892 30628 tablet_bootstrap.cc:492] T 8bb679bf525a4a2ebafeb73740230569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:17:55.506120 30628 ts_tablet_manager.cc:1367] T 8bb679bf525a4a2ebafeb73740230569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:17:55.506194 30852 tablet_bootstrap.cc:492] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:55.506371 30852 ts_tablet_manager.cc:1367] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 08:17:55.507046 30628 raft_consensus.cc:348] T 8bb679bf525a4a2ebafeb73740230569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:55.507278 30628 raft_consensus.cc:374] T 8bb679bf525a4a2ebafeb73740230569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:55.507354 30628 raft_consensus.cc:729] T 8bb679bf525a4a2ebafeb73740230569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:17:55.507148 30852 raft_consensus.cc:348] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:55.507540 30852 raft_consensus.cc:374] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:55.507674 30852 raft_consensus.cc:729] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:55.507609 30628 consensus_queue.cc:260] T 8bb679bf525a4a2ebafeb73740230569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:55.508209 30628 ts_tablet_manager.cc:1398] T 8bb679bf525a4a2ebafeb73740230569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:55.508109 30852 consensus_queue.cc:260] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:55.508740 30852 ts_tablet_manager.cc:1398] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:55.512841 30851 tablet_bootstrap.cc:492] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:55.514227 30851 tablet_bootstrap.cc:654] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.515923 30851 tablet_bootstrap.cc:492] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:17:55.516098 30851 ts_tablet_manager.cc:1367] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240502 08:17:55.516862 30851 raft_consensus.cc:348] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:55.517079 30851 raft_consensus.cc:374] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:55.517161 30851 raft_consensus.cc:729] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:17:55.517168 27113 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet b2aa620b1bd747a79ec84d2cf7b727ef (table rebalancer_dynamic_table [id=046e5e3d46974da29f94c4f9d2803571]). Expected version 1 got 0
I20240502 08:17:55.517388 30851 consensus_queue.cc:260] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:55.518086 30851 ts_tablet_manager.cc:1398] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:55.524636 30848 tablet.cc:1722] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:55.525179 30848 tablet.cc:1620] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1: MemRowSet was empty: no flush needed.
I20240502 08:17:55.525497 27114 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=046e5e3d46974da29f94c4f9d2803571] alter complete (version 1)
I20240502 08:17:55.525857 27258 consensus_queue.cc:237] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:17:55.527596 30823 tablet_bootstrap.cc:492] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:55.529531 27114 catalog_manager.cc:5576] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291) does not have the latest schema for tablet b2aa620b1bd747a79ec84d2cf7b727ef (table rebalancer_dynamic_table [id=046e5e3d46974da29f94c4f9d2803571]). Expected version 1 got 0
I20240502 08:17:55.530789 30823 ts_tablet_manager.cc:1367] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.004s
I20240502 08:17:55.531833 30823 raft_consensus.cc:348] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:55.532176 30823 raft_consensus.cc:374] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:55.533177 30823 raft_consensus.cc:729] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:55.534198 30823 consensus_queue.cc:260] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:55.535847 30823 ts_tablet_manager.cc:1398] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:17:55.538724 30823 tablet_bootstrap.cc:492] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:55.540114 27114 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet b2aa620b1bd747a79ec84d2cf7b727ef (table rebalancer_dynamic_table [id=046e5e3d46974da29f94c4f9d2803571]). Expected version 1 got 0
I20240502 08:17:55.540334 30823 tablet_bootstrap.cc:654] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.550207 27388 raft_consensus.cc:1261] T 1662e908146944fd8e74d901e497e35d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:55.550201 27521 raft_consensus.cc:1261] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:55.551074 30794 consensus_queue.cc:1035] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:55.551714 30865 tablet.cc:1722] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:55.552856 30865 tablet.cc:1620] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:17:55.554718 30795 consensus_queue.cc:1035] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:55.558808 30795 raft_consensus.cc:2932] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:17:55.564267 27114 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet b2aa620b1bd747a79ec84d2cf7b727ef (table rebalancer_dynamic_table [id=046e5e3d46974da29f94c4f9d2803571]). Expected version 1 got 0
W20240502 08:17:55.565021 27191 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Couldn't send request to peer 1f0bf4bc3c5146539146b825b048bff3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1662e908146944fd8e74d901e497e35d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:55.565948 30823 tablet_bootstrap.cc:492] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:55.565609 27390 raft_consensus.cc:2932] T 1662e908146944fd8e74d901e497e35d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:17:55.566124 30823 ts_tablet_manager.cc:1367] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.010s
I20240502 08:17:55.567860 30823 raft_consensus.cc:348] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:55.568070 30823 raft_consensus.cc:374] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:55.568166 30823 raft_consensus.cc:729] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:55.568416 30823 consensus_queue.cc:260] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:55.568919 30823 ts_tablet_manager.cc:1398] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:55.569571 30823 tablet_bootstrap.cc:492] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:55.570034 30856 raft_consensus.cc:482] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:55.570190 30856 raft_consensus.cc:504] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:55.570395 27113 catalog_manager.cc:5484] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:55.570919 30856 leader_election.cc:288] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:55.571247 30823 tablet_bootstrap.cc:654] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.571379 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83993ff2073842988937597dd3999d0d" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:17:55.571583 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83993ff2073842988937597dd3999d0d" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:17:55.571785 30160 raft_consensus.cc:2454] T 83993ff2073842988937597dd3999d0d P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
W20240502 08:17:55.572252 30355 leader_election.cc:341] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:55.572458 30355 leader_election.cc:302] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters: 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:17:55.572926 30856 raft_consensus.cc:2781] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:55.573022 30856 raft_consensus.cc:482] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:55.573098 30856 raft_consensus.cc:3037] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:55.573364 30823 tablet_bootstrap.cc:492] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:55.573493 30823 ts_tablet_manager.cc:1367] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:55.574654 30823 raft_consensus.cc:348] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:55.575246 30823 raft_consensus.cc:374] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:55.575315 30823 raft_consensus.cc:729] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:55.575528 30823 consensus_queue.cc:260] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:55.575691 30856 raft_consensus.cc:504] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:55.576165 30823 ts_tablet_manager.cc:1398] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:55.575670 27521 raft_consensus.cc:2932] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:17:55.576706 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83993ff2073842988937597dd3999d0d" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:17:55.576884 30160 raft_consensus.cc:3037] T 83993ff2073842988937597dd3999d0d P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:55.578444 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83993ff2073842988937597dd3999d0d" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:17:55.578661 30028 raft_consensus.cc:3037] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:55.579587 30160 raft_consensus.cc:2454] T 83993ff2073842988937597dd3999d0d P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:17:55.581331 30028 raft_consensus.cc:2454] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:17:55.583849 30358 leader_election.cc:302] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:55.584113 30627 raft_consensus.cc:2781] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:55.587453 30856 leader_election.cc:288] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:55.592846 30627 raft_consensus.cc:686] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:17:55.593154 30627 consensus_queue.cc:237] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:55.596601 27107 catalog_manager.cc:5484] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:55.602717 30829 raft_consensus.cc:482] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:55.602870 30829 raft_consensus.cc:504] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:55.603515 30829 leader_election.cc:288] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:17:55.604303 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb679bf525a4a2ebafeb73740230569" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:17:55.604557 30160 raft_consensus.cc:2454] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:17:55.604578 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb679bf525a4a2ebafeb73740230569" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:17:55.604760 30422 raft_consensus.cc:2454] T 8bb679bf525a4a2ebafeb73740230569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:17:55.605165 29961 leader_election.cc:302] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters:
I20240502 08:17:55.605753 30829 raft_consensus.cc:2781] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:55.605825 30829 raft_consensus.cc:482] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:55.605886 30829 raft_consensus.cc:3037] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:55.606853 27521 consensus_queue.cc:237] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:17:55.607754 30829 raft_consensus.cc:504] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:55.608717 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb679bf525a4a2ebafeb73740230569" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:17:55.608783 30829 leader_election.cc:288] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:17:55.608935 30160 raft_consensus.cc:3037] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:55.609633 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb679bf525a4a2ebafeb73740230569" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:17:55.609833 30422 raft_consensus.cc:3037] T 8bb679bf525a4a2ebafeb73740230569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:55.611737 30160 raft_consensus.cc:2454] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:17:55.612500 30422 raft_consensus.cc:2454] T 8bb679bf525a4a2ebafeb73740230569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:17:55.613000 29961 leader_election.cc:302] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters:
I20240502 08:17:55.613790 30829 raft_consensus.cc:2781] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:55.614506 30829 raft_consensus.cc:686] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:17:55.614777 30829 consensus_queue.cc:237] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:55.617852 27107 catalog_manager.cc:5484] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:55.619127 27390 raft_consensus.cc:1261] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:55.619189 27258 raft_consensus.cc:1261] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:55.620927 30335 consensus_queue.cc:1035] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:55.621532 30335 consensus_queue.cc:1035] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:55.624715 30335 raft_consensus.cc:2932] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:17:55.625352 27258 raft_consensus.cc:2932] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:17:55.632658 27107 catalog_manager.cc:5484] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:17:55.637791 27107 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:17:55.651335 30848 tablet.cc:1722] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:55.651973 30848 tablet.cc:1620] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1: MemRowSet was empty: no flush needed.
W20240502 08:17:55.654822 27456 consensus_peers.cc:485] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 -> Peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Couldn't send request to peer 84cda20738d2452da166e31fc6da57a6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db55b42c724246da8d9cb23369826279. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:55.657328 27390 raft_consensus.cc:2932] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:17:55.658813 30865 tablet.cc:1722] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:55.659332 30865 tablet.cc:1620] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:17:55.661096 30849 tablet.cc:1722] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:55.661509 30849 tablet.cc:1620] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:17:55.666385 27114 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 08:17:55.666581 27114 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 08:17:55.671453 27113 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=046e5e3d46974da29f94c4f9d2803571] alter complete (version 2)
I20240502 08:17:55.679785 27114 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet b57e9fcac6354a2d9d52dbd299241bd4
I20240502 08:17:55.680388 27114 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 2643aca45d404250b6d56ca12317290f
I20240502 08:17:55.680887 27114 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 1c75d776e2b34e369111d7803d1b3491
I20240502 08:17:55.682250 27237 tablet_service.cc:1514] Processing DeleteTablet for tablet b57e9fcac6354a2d9d52dbd299241bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:55.682430 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet 2643aca45d404250b6d56ca12317290f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:55.682559 30524 tablet_replica.cc:330] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:55.682766 30524 raft_consensus.cc:2229] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:55.683053 30524 raft_consensus.cc:2258] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.683130 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c75d776e2b34e369111d7803d1b3491 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:17:55.683655 27499 tablet_service.cc:1514] Processing DeleteTablet for tablet b57e9fcac6354a2d9d52dbd299241bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:17:55.683903 30524 ts_tablet_manager.cc:1875] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.683979 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet 2643aca45d404250b6d56ca12317290f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:55.684306 30522 tablet_replica.cc:330] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:55.684334 27629 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c75d776e2b34e369111d7803d1b3491 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:55.684504 30522 raft_consensus.cc:2229] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:55.684736 30522 raft_consensus.cc:2258] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.685354 30522 ts_tablet_manager.cc:1875] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.693531 30522 ts_tablet_manager.cc:1888] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:55.693655 30522 log.cc:1196] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/2643aca45d404250b6d56ca12317290f
I20240502 08:17:55.694074 30522 ts_tablet_manager.cc:1909] T 2643aca45d404250b6d56ca12317290f P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:55.695053 30522 tablet_replica.cc:330] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:55.695098 30524 ts_tablet_manager.cc:1888] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:55.695192 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 2643aca45d404250b6d56ca12317290f (table rebalance_test_table_12 [id=d1443c13acb24eacba47f4561d2d0443]) successfully deleted
I20240502 08:17:55.695267 30522 raft_consensus.cc:2229] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:55.695411 30524 log.cc:1196] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/b57e9fcac6354a2d9d52dbd299241bd4
I20240502 08:17:55.695472 30522 raft_consensus.cc:2258] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.695915 30524 ts_tablet_manager.cc:1909] T b57e9fcac6354a2d9d52dbd299241bd4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:55.696103 30522 ts_tablet_manager.cc:1875] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.697162 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet b57e9fcac6354a2d9d52dbd299241bd4 (table rebalance_test_table_12 [id=d1443c13acb24eacba47f4561d2d0443]) successfully deleted
I20240502 08:17:55.697993 30524 tablet_replica.cc:330] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:55.698359 30524 raft_consensus.cc:2229] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:55.698694 30524 raft_consensus.cc:2258] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.701110 30524 ts_tablet_manager.cc:1875] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.705085 30896 tablet_replica.cc:330] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:17:55.705293 30522 ts_tablet_manager.cc:1888] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:55.705303 30896 raft_consensus.cc:2229] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:55.705451 30522 log.cc:1196] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/1c75d776e2b34e369111d7803d1b3491
I20240502 08:17:55.705852 30522 ts_tablet_manager.cc:1909] T 1c75d776e2b34e369111d7803d1b3491 P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
W20240502 08:17:55.706086 27586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.2:46291, user_credentials={real_user=slave}} blocked reactor thread for 63986.8us
I20240502 08:17:55.706414 30896 raft_consensus.cc:2258] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.706979 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 1c75d776e2b34e369111d7803d1b3491 (table rebalance_test_table_12 [id=d1443c13acb24eacba47f4561d2d0443]) successfully deleted
I20240502 08:17:55.707938 30896 ts_tablet_manager.cc:1875] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.714951 30896 ts_tablet_manager.cc:1888] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:55.715149 30896 log.cc:1196] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/1c75d776e2b34e369111d7803d1b3491
I20240502 08:17:55.715637 30896 ts_tablet_manager.cc:1909] T 1c75d776e2b34e369111d7803d1b3491 P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:17:55.716437 30524 ts_tablet_manager.cc:1888] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:55.716719 30896 tablet_replica.cc:330] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:17:55.718312 27369 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c75d776e2b34e369111d7803d1b3491 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:55.718356 27368 tablet_service.cc:1514] Processing DeleteTablet for tablet 2643aca45d404250b6d56ca12317290f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:55.718312 27370 tablet_service.cc:1514] Processing DeleteTablet for tablet b57e9fcac6354a2d9d52dbd299241bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:55.719131 30523 tablet_replica.cc:330] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:55.719375 30523 raft_consensus.cc:2229] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:55.719686 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 1c75d776e2b34e369111d7803d1b3491 (table rebalance_test_table_12 [id=d1443c13acb24eacba47f4561d2d0443]) successfully deleted
I20240502 08:17:55.719753 30523 raft_consensus.cc:2258] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.720389 30523 ts_tablet_manager.cc:1875] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.720618 30524 log.cc:1196] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/2643aca45d404250b6d56ca12317290f
I20240502 08:17:55.721103 30524 ts_tablet_manager.cc:1909] T 2643aca45d404250b6d56ca12317290f P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:55.722177 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet 2643aca45d404250b6d56ca12317290f (table rebalance_test_table_12 [id=d1443c13acb24eacba47f4561d2d0443]) successfully deleted
I20240502 08:17:55.726431 30523 ts_tablet_manager.cc:1888] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:55.726574 30523 log.cc:1196] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/1c75d776e2b34e369111d7803d1b3491
I20240502 08:17:55.726897 30523 ts_tablet_manager.cc:1909] T 1c75d776e2b34e369111d7803d1b3491 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:55.727679 30523 tablet_replica.cc:330] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:55.727701 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 1c75d776e2b34e369111d7803d1b3491 (table rebalance_test_table_12 [id=d1443c13acb24eacba47f4561d2d0443]) successfully deleted
I20240502 08:17:55.727878 30523 raft_consensus.cc:2229] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:55.728175 30523 raft_consensus.cc:2258] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.728796 30523 ts_tablet_manager.cc:1875] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.729121 27390 consensus_queue.cc:237] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
W20240502 08:17:55.731906 27325 consensus_peers.cc:485] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): Couldn't send request to peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b57e9fcac6354a2d9d52dbd299241bd4. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:55.732147 27326 consensus_peers.cc:485] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:55.732226 27323 consensus_peers.cc:485] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Couldn't send request to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b57e9fcac6354a2d9d52dbd299241bd4. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:55.717005 30896 raft_consensus.cc:2229] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:55.732558 30896 raft_consensus.cc:2258] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.733103 30896 ts_tablet_manager.cc:1875] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.741207 30523 ts_tablet_manager.cc:1888] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:55.741322 30523 log.cc:1196] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/2643aca45d404250b6d56ca12317290f
I20240502 08:17:55.741675 30523 ts_tablet_manager.cc:1909] T 2643aca45d404250b6d56ca12317290f P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:55.742432 30896 ts_tablet_manager.cc:1888] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:55.742558 30896 log.cc:1196] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/b57e9fcac6354a2d9d52dbd299241bd4
I20240502 08:17:55.742954 30896 ts_tablet_manager.cc:1909] T b57e9fcac6354a2d9d52dbd299241bd4 P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:17:55.743867 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet b57e9fcac6354a2d9d52dbd299241bd4 (table rebalance_test_table_12 [id=d1443c13acb24eacba47f4561d2d0443]) successfully deleted
I20240502 08:17:55.744189 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 2643aca45d404250b6d56ca12317290f (table rebalance_test_table_12 [id=d1443c13acb24eacba47f4561d2d0443]) successfully deleted
I20240502 08:17:55.744510 30523 tablet_replica.cc:330] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:55.744724 30523 raft_consensus.cc:2229] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:55.744990 30523 pending_rounds.cc:62] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1: Trying to abort 1 pending ops.
I20240502 08:17:55.745128 30523 pending_rounds.cc:68] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1: Aborting op as it isn't in flight: id { term: 1 index: 4 } timestamp: 7023156738984456192 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "b57e9fcac6354a2d9d52dbd299241bd4" old_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } } new_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } } }
I20240502 08:17:55.745512 30523 raft_consensus.cc:2894] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Aborting config change with OpId 1.4: Aborted: Op aborted
I20240502 08:17:55.745654 30523 raft_consensus.cc:2866] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 08:17:55.746096 30523 raft_consensus.cc:2258] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.747108 30523 ts_tablet_manager.cc:1875] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.753829 30523 ts_tablet_manager.cc:1888] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:55.753993 30523 log.cc:1196] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/b57e9fcac6354a2d9d52dbd299241bd4
I20240502 08:17:55.754781 30523 ts_tablet_manager.cc:1909] T b57e9fcac6354a2d9d52dbd299241bd4 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:55.757581 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet b57e9fcac6354a2d9d52dbd299241bd4 (table rebalance_test_table_12 [id=d1443c13acb24eacba47f4561d2d0443]) successfully deleted
I20240502 08:17:55.773567 27114 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:17:55.775628 30826 raft_consensus.cc:482] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:55.775826 30826 raft_consensus.cc:504] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:55.776719 30826 leader_election.cc:288] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:17:55.777757 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63eec7e63fe4e71b1958903e0d2cb00" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:17:55.778082 30027 raft_consensus.cc:2454] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:55.800217 27323 leader_election.cc:302] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:55.800496 30826 raft_consensus.cc:2781] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:55.800590 30826 raft_consensus.cc:482] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:55.800689 30826 raft_consensus.cc:3037] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:55.803575 30826 raft_consensus.cc:504] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:55.805409 30826 leader_election.cc:288] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:17:55.805970 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63eec7e63fe4e71b1958903e0d2cb00" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:17:55.806227 30028 raft_consensus.cc:3037] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:55.807118 30848 tablet.cc:1722] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:55.808990 30028 raft_consensus.cc:2454] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:55.809363 30849 tablet.cc:1722] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:55.810245 30865 tablet.cc:1722] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:55.810452 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63eec7e63fe4e71b1958903e0d2cb00" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:17:55.810611 27323 leader_election.cc:302] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:55.810667 30288 raft_consensus.cc:2454] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:55.810828 30826 raft_consensus.cc:2781] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:55.810979 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63eec7e63fe4e71b1958903e0d2cb00" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:17:55.811055 30826 raft_consensus.cc:686] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:17:55.811163 30290 raft_consensus.cc:3037] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:55.811305 30826 consensus_queue.cc:237] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:55.815716 30290 raft_consensus.cc:2454] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:55.818158 27114 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=046e5e3d46974da29f94c4f9d2803571] alter complete (version 3)
I20240502 08:17:55.822376 27113 catalog_manager.cc:5484] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:55.921900 27110 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:17:55.922199 27110 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:17:55.925446 27110 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet b2aa620b1bd747a79ec84d2cf7b727ef
I20240502 08:17:55.926246 27370 tablet_service.cc:1514] Processing DeleteTablet for tablet b2aa620b1bd747a79ec84d2cf7b727ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:55.926726 30523 tablet_replica.cc:330] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:55.926980 30523 raft_consensus.cc:2229] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:55.927151 30401 tablet_service.cc:1514] Processing DeleteTablet for tablet b2aa620b1bd747a79ec84d2cf7b727ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:56694
I20240502 08:17:55.927390 30523 raft_consensus.cc:2258] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.928236 30008 tablet_service.cc:1514] Processing DeleteTablet for tablet b2aa620b1bd747a79ec84d2cf7b727ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:55 UTC) from {username='slave'} at 127.0.0.1:58374
I20240502 08:17:55.929144 30523 ts_tablet_manager.cc:1875] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.939251 30523 ts_tablet_manager.cc:1888] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:55.945613 30523 log.cc:1196] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/b2aa620b1bd747a79ec84d2cf7b727ef
I20240502 08:17:55.946305 30523 ts_tablet_manager.cc:1909] T b2aa620b1bd747a79ec84d2cf7b727ef P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:55.947911 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet b2aa620b1bd747a79ec84d2cf7b727ef (table rebalancer_dynamic_table_new_name [id=046e5e3d46974da29f94c4f9d2803571]) successfully deleted
I20240502 08:17:55.951710 30937 tablet_replica.cc:330] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:17:55.951916 30937 raft_consensus.cc:2229] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:55.952112 30937 raft_consensus.cc:2258] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.952824 30937 ts_tablet_manager.cc:1875] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.957589 30937 ts_tablet_manager.cc:1888] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:55.958106 30936 tablet_replica.cc:330] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:17:55.958381 30936 raft_consensus.cc:2229] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:55.958678 30936 raft_consensus.cc:2258] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:55.959692 30936 ts_tablet_manager.cc:1875] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:55.962100 30937 log.cc:1196] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/b2aa620b1bd747a79ec84d2cf7b727ef
I20240502 08:17:55.962513 30937 ts_tablet_manager.cc:1909] T b2aa620b1bd747a79ec84d2cf7b727ef P 1f0bf4bc3c5146539146b825b048bff3: Deleting consensus metadata
I20240502 08:17:55.964450 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet b2aa620b1bd747a79ec84d2cf7b727ef (table rebalancer_dynamic_table_new_name [id=046e5e3d46974da29f94c4f9d2803571]) successfully deleted
I20240502 08:17:55.966233 30936 ts_tablet_manager.cc:1888] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:55.966394 30936 log.cc:1196] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/b2aa620b1bd747a79ec84d2cf7b727ef
I20240502 08:17:55.966830 30936 ts_tablet_manager.cc:1909] T b2aa620b1bd747a79ec84d2cf7b727ef P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting consensus metadata
I20240502 08:17:55.969537 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet b2aa620b1bd747a79ec84d2cf7b727ef (table rebalancer_dynamic_table_new_name [id=046e5e3d46974da29f94c4f9d2803571]) successfully deleted
I20240502 08:17:55.971534 27238 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:17:55.981562 30008 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:55.998080 30268 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:56.002547 30829 consensus_queue.cc:1035] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:17:56.003417 30139 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:56.015779 30401 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:56.017611 30627 consensus_queue.cc:1035] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:56.026397 30874 consensus_queue.cc:1035] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:56.028004 27116 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:17:56.038313 27500 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:17:56.040148 30829 consensus_queue.cc:1035] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:56.042969 30139 tablet_service.cc:1467] Processing CreateTablet for tablet f2d13299ba914a348fbcabef218c57eb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=319468b7ec92492795908f84cc07ff24]), partition=
I20240502 08:17:56.043442 30139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2d13299ba914a348fbcabef218c57eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:56.066844 30401 tablet_service.cc:1467] Processing CreateTablet for tablet f2d13299ba914a348fbcabef218c57eb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=319468b7ec92492795908f84cc07ff24]), partition=
I20240502 08:17:56.066897 30268 tablet_service.cc:1467] Processing CreateTablet for tablet f2d13299ba914a348fbcabef218c57eb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=319468b7ec92492795908f84cc07ff24]), partition=
I20240502 08:17:56.067250 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2d13299ba914a348fbcabef218c57eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:56.067234 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2d13299ba914a348fbcabef218c57eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:56.071736 30957 ts_tablet_manager.cc:902] T 1662e908146944fd8e74d901e497e35d P 1f0bf4bc3c5146539146b825b048bff3: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:56.074096 30958 tablet_bootstrap.cc:492] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:17:56.078088 30958 tablet_bootstrap.cc:654] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:56.087862 30958 tablet_bootstrap.cc:492] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:17:56.088045 30958 ts_tablet_manager.cc:1367] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240502 08:17:56.089027 30958 raft_consensus.cc:348] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:56.089303 30958 raft_consensus.cc:374] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:56.089422 30958 raft_consensus.cc:729] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:17:56.092754 30958 consensus_queue.cc:260] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:56.094687 30958 ts_tablet_manager.cc:1398] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:17:56.095260 27370 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:17:56.101084 30957 tablet_copy_client.cc:308] T 1662e908146944fd8e74d901e497e35d P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:17:56.108668 30956 tablet_bootstrap.cc:492] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:56.110443 30956 tablet_bootstrap.cc:654] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:56.112999 30960 tablet_bootstrap.cc:492] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:56.114552 30960 tablet_bootstrap.cc:654] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:56.124470 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet 1662e908146944fd8e74d901e497e35d from peer 1f0bf4bc3c5146539146b825b048bff3 ({username='slave'} at 127.20.207.5:51633)
I20240502 08:17:56.124740 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet 1662e908146944fd8e74d901e497e35d from peer 1f0bf4bc3c5146539146b825b048bff3 at {username='slave'} at 127.20.207.5:51633: session id = 1f0bf4bc3c5146539146b825b048bff3-1662e908146944fd8e74d901e497e35d
I20240502 08:17:56.129658 27278 tablet_copy_source_session.cc:205] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:56.134508 30957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1662e908146944fd8e74d901e497e35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:56.145380 30957 tablet_copy_client.cc:783] T 1662e908146944fd8e74d901e497e35d P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:56.145815 30957 tablet_copy_client.cc:647] T 1662e908146944fd8e74d901e497e35d P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:56.158195 30960 tablet_bootstrap.cc:492] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:56.158851 30960 ts_tablet_manager.cc:1367] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20240502 08:17:56.159271 30956 tablet_bootstrap.cc:492] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:17:56.159461 30956 ts_tablet_manager.cc:1367] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20240502 08:17:56.160287 30960 raft_consensus.cc:348] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:56.160576 30960 raft_consensus.cc:374] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:56.160712 30960 raft_consensus.cc:729] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:56.166257 30957 tablet_copy_client.cc:523] T 1662e908146944fd8e74d901e497e35d P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:56.170784 30953 ts_tablet_manager.cc:902] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:56.161072 30960 consensus_queue.cc:260] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:56.160454 30956 raft_consensus.cc:348] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:56.172473 30956 raft_consensus.cc:374] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:56.172622 30956 raft_consensus.cc:729] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:17:56.172943 30956 consensus_queue.cc:260] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:56.173760 30960 ts_tablet_manager.cc:1398] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 08:17:56.175302 30957 tablet_bootstrap.cc:492] T 1662e908146944fd8e74d901e497e35d P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:56.177690 30956 ts_tablet_manager.cc:1398] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20240502 08:17:56.178269 27629 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:56.186941 27110 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 08:17:56.187172 27110 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 08:17:56.191669 27110 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 9935a8efe788429dbdfc8c61387adb60
I20240502 08:17:56.194595 27629 tablet_service.cc:1514] Processing DeleteTablet for tablet 9935a8efe788429dbdfc8c61387adb60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:56.194610 27110 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet b613c84f191b4dfa9b306fa605456b95
I20240502 08:17:56.194850 30522 tablet_replica.cc:330] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:56.195044 30522 raft_consensus.cc:2229] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:56.195307 30522 raft_consensus.cc:2258] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.195780 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet b613c84f191b4dfa9b306fa605456b95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:17:56.195834 27370 tablet_service.cc:1514] Processing DeleteTablet for tablet 9935a8efe788429dbdfc8c61387adb60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:56.195991 30522 ts_tablet_manager.cc:1875] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.196144 30896 tablet_replica.cc:330] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:17:56.196429 30896 raft_consensus.cc:2229] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:56.196800 30896 raft_consensus.cc:2258] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.197110 27629 tablet_service.cc:1514] Processing DeleteTablet for tablet b613c84f191b4dfa9b306fa605456b95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:56.197487 30896 ts_tablet_manager.cc:1875] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.198125 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet 9935a8efe788429dbdfc8c61387adb60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:56.198403 30524 tablet_replica.cc:330] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:56.198416 27368 tablet_service.cc:1514] Processing DeleteTablet for tablet b613c84f191b4dfa9b306fa605456b95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:56.198581 30524 raft_consensus.cc:2229] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:56.198947 30524 raft_consensus.cc:2258] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.199591 30524 ts_tablet_manager.cc:1875] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.202155 30523 tablet_replica.cc:330] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:56.202404 30523 raft_consensus.cc:2229] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:56.202661 30523 raft_consensus.cc:2258] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.202759 30522 ts_tablet_manager.cc:1888] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:56.202853 30522 log.cc:1196] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/9935a8efe788429dbdfc8c61387adb60
I20240502 08:17:56.203235 30522 ts_tablet_manager.cc:1909] T 9935a8efe788429dbdfc8c61387adb60 P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:56.203331 30523 ts_tablet_manager.cc:1875] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.204202 30522 tablet_replica.cc:330] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:56.204398 30522 raft_consensus.cc:2229] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:56.204560 30522 raft_consensus.cc:2258] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.205152 30522 ts_tablet_manager.cc:1875] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.206487 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 9935a8efe788429dbdfc8c61387adb60 (table rebalance_test_table_14 [id=ecb31f1a638e457fa80fdec831562539]) successfully deleted
I20240502 08:17:56.212056 30523 ts_tablet_manager.cc:1888] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:56.212260 30523 log.cc:1196] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/9935a8efe788429dbdfc8c61387adb60
I20240502 08:17:56.212751 30523 ts_tablet_manager.cc:1909] T 9935a8efe788429dbdfc8c61387adb60 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:56.213778 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 9935a8efe788429dbdfc8c61387adb60 (table rebalance_test_table_14 [id=ecb31f1a638e457fa80fdec831562539]) successfully deleted
I20240502 08:17:56.215698 30524 ts_tablet_manager.cc:1888] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:56.215816 30524 log.cc:1196] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/9935a8efe788429dbdfc8c61387adb60
I20240502 08:17:56.216217 30524 ts_tablet_manager.cc:1909] T 9935a8efe788429dbdfc8c61387adb60 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:56.217134 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet 9935a8efe788429dbdfc8c61387adb60 (table rebalance_test_table_14 [id=ecb31f1a638e457fa80fdec831562539]) successfully deleted
I20240502 08:17:56.216970 30896 ts_tablet_manager.cc:1888] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:56.217473 30896 log.cc:1196] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/b613c84f191b4dfa9b306fa605456b95
I20240502 08:17:56.217981 30896 ts_tablet_manager.cc:1909] T b613c84f191b4dfa9b306fa605456b95 P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:17:56.219095 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet b613c84f191b4dfa9b306fa605456b95 (table rebalance_test_table_14 [id=ecb31f1a638e457fa80fdec831562539]) successfully deleted
I20240502 08:17:56.220029 30826 consensus_queue.cc:1035] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:56.228492 30856 raft_consensus.cc:482] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:56.228672 30856 raft_consensus.cc:504] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:56.228842 30522 ts_tablet_manager.cc:1888] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:56.228957 30522 log.cc:1196] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/b613c84f191b4dfa9b306fa605456b95
I20240502 08:17:56.229321 30522 ts_tablet_manager.cc:1909] T b613c84f191b4dfa9b306fa605456b95 P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:56.229379 30856 leader_election.cc:288] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:17:56.230103 30158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2d13299ba914a348fbcabef218c57eb" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:17:56.230320 30158 raft_consensus.cc:2454] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:17:56.230475 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2d13299ba914a348fbcabef218c57eb" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:17:56.230695 30290 raft_consensus.cc:2454] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:17:56.230710 30358 leader_election.cc:302] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:56.230986 30856 raft_consensus.cc:2781] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:56.231079 30856 raft_consensus.cc:482] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:56.231148 30856 raft_consensus.cc:3037] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.233559 30523 tablet_replica.cc:330] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:56.233695 30856 raft_consensus.cc:504] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:56.234123 30523 raft_consensus.cc:2229] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:56.234730 30158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2d13299ba914a348fbcabef218c57eb" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:17:56.234915 30158 raft_consensus.cc:3037] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.234942 30523 raft_consensus.cc:2258] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.235616 30523 ts_tablet_manager.cc:1875] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.237591 30158 raft_consensus.cc:2454] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:17:56.238458 30358 leader_election.cc:302] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:56.238746 30874 raft_consensus.cc:2781] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:56.238863 30874 raft_consensus.cc:686] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:17:56.239090 30874 consensus_queue.cc:237] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:56.240664 27110 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet dd7732b2d14c4072b64fa290f1c613ab
I20240502 08:17:56.241312 27629 tablet_service.cc:1514] Processing DeleteTablet for tablet dd7732b2d14c4072b64fa290f1c613ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:56.242040 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2d13299ba914a348fbcabef218c57eb" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:17:56.242233 30290 raft_consensus.cc:3037] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.244824 30290 raft_consensus.cc:2454] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:17:56.245354 30856 leader_election.cc:288] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:17:56.245990 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet dd7732b2d14c4072b64fa290f1c613ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:56.246307 30524 tablet_replica.cc:330] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:56.246595 30524 raft_consensus.cc:2229] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:56.246954 30524 raft_consensus.cc:2258] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.247471 27368 tablet_service.cc:1514] Processing DeleteTablet for tablet dd7732b2d14c4072b64fa290f1c613ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:56.247795 30524 ts_tablet_manager.cc:1875] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.254123 30957 tablet_bootstrap.cc:492] T 1662e908146944fd8e74d901e497e35d P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:56.254710 30957 tablet_bootstrap.cc:492] T 1662e908146944fd8e74d901e497e35d P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap complete.
I20240502 08:17:56.254971 30957 ts_tablet_manager.cc:1367] T 1662e908146944fd8e74d901e497e35d P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.080s user 0.045s sys 0.022s
I20240502 08:17:56.257377 30957 raft_consensus.cc:348] T 1662e908146944fd8e74d901e497e35d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:17:56.257664 30957 raft_consensus.cc:729] T 1662e908146944fd8e74d901e497e35d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: LEARNER
I20240502 08:17:56.258374 30953 tablet_copy_client.cc:308] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:17:56.258299 30957 consensus_queue.cc:260] T 1662e908146944fd8e74d901e497e35d P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:17:56.259034 30957 ts_tablet_manager.cc:1398] T 1662e908146944fd8e74d901e497e35d P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:17:56.259889 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session 1f0bf4bc3c5146539146b825b048bff3-1662e908146944fd8e74d901e497e35d received from {username='slave'} at 127.20.207.5:51633
I20240502 08:17:56.260071 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session 1f0bf4bc3c5146539146b825b048bff3-1662e908146944fd8e74d901e497e35d on tablet 1662e908146944fd8e74d901e497e35d with peer 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:17:56.260686 30524 ts_tablet_manager.cc:1888] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:56.260828 30524 log.cc:1196] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/dd7732b2d14c4072b64fa290f1c613ab
I20240502 08:17:56.261368 30524 ts_tablet_manager.cc:1909] T dd7732b2d14c4072b64fa290f1c613ab P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:56.262902 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet dd7732b2d14c4072b64fa290f1c613ab (table rebalance_test_table_14 [id=ecb31f1a638e457fa80fdec831562539]) successfully deleted
I20240502 08:17:56.270800 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet db55b42c724246da8d9cb23369826279 from peer 84cda20738d2452da166e31fc6da57a6 ({username='slave'} at 127.20.207.6:40325)
I20240502 08:17:56.270985 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet db55b42c724246da8d9cb23369826279 from peer 84cda20738d2452da166e31fc6da57a6 at {username='slave'} at 127.20.207.6:40325: session id = 84cda20738d2452da166e31fc6da57a6-db55b42c724246da8d9cb23369826279
I20240502 08:17:56.275785 27541 tablet_copy_source_session.cc:205] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:56.280679 30953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db55b42c724246da8d9cb23369826279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:56.284317 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet b613c84f191b4dfa9b306fa605456b95 (table rebalance_test_table_14 [id=ecb31f1a638e457fa80fdec831562539]) successfully deleted
I20240502 08:17:56.284513 30522 tablet_replica.cc:330] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:56.284727 30522 raft_consensus.cc:2229] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:56.284942 30522 raft_consensus.cc:2258] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.288022 30522 ts_tablet_manager.cc:1875] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.311097 30522 ts_tablet_manager.cc:1888] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:56.313246 30762 consensus_queue.cc:1035] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:56.314424 30522 log.cc:1196] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/dd7732b2d14c4072b64fa290f1c613ab
I20240502 08:17:56.314940 30522 ts_tablet_manager.cc:1909] T dd7732b2d14c4072b64fa290f1c613ab P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:56.316982 27110 catalog_manager.cc:5484] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:56.318290 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet dd7732b2d14c4072b64fa290f1c613ab (table rebalance_test_table_14 [id=ecb31f1a638e457fa80fdec831562539]) successfully deleted
I20240502 08:17:56.332741 30523 ts_tablet_manager.cc:1888] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:56.332958 30523 log.cc:1196] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/b613c84f191b4dfa9b306fa605456b95
I20240502 08:17:56.333509 30523 ts_tablet_manager.cc:1909] T b613c84f191b4dfa9b306fa605456b95 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:56.334760 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet b613c84f191b4dfa9b306fa605456b95 (table rebalance_test_table_14 [id=ecb31f1a638e457fa80fdec831562539]) successfully deleted
I20240502 08:17:56.336045 30523 tablet_replica.cc:330] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:56.336309 30523 raft_consensus.cc:2229] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:56.336606 30523 raft_consensus.cc:2258] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.337478 30523 ts_tablet_manager.cc:1875] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.361173 30953 tablet_copy_client.cc:783] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:56.361558 30953 tablet_copy_client.cc:647] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:56.363983 30953 tablet_copy_client.cc:523] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:56.373795 30953 tablet_bootstrap.cc:492] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:56.398393 30523 ts_tablet_manager.cc:1888] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:56.398564 30523 log.cc:1196] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/dd7732b2d14c4072b64fa290f1c613ab
I20240502 08:17:56.399003 30523 ts_tablet_manager.cc:1909] T dd7732b2d14c4072b64fa290f1c613ab P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:56.400041 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet dd7732b2d14c4072b64fa290f1c613ab (table rebalance_test_table_14 [id=ecb31f1a638e457fa80fdec831562539]) successfully deleted
I20240502 08:17:56.420071 27110 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:56.430569 30008 tablet_service.cc:1467] Processing CreateTablet for tablet 387e9ef5899541b0be27a41dbcf4475c (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=05e7c365fc854833a30f18097c117d10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:56.431030 30008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387e9ef5899541b0be27a41dbcf4475c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:56.431700 27629 tablet_service.cc:1467] Processing CreateTablet for tablet ec6c977cac054b758a43d846e821707a (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=05e7c365fc854833a30f18097c117d10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:56.432085 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6c977cac054b758a43d846e821707a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:56.435338 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 201d79895c2741dd9d9de0cc9f605a91 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=05e7c365fc854833a30f18097c117d10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:56.435353 27500 tablet_service.cc:1467] Processing CreateTablet for tablet 387e9ef5899541b0be27a41dbcf4475c (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=05e7c365fc854833a30f18097c117d10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:56.435724 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 201d79895c2741dd9d9de0cc9f605a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:56.435772 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387e9ef5899541b0be27a41dbcf4475c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:56.438702 30139 tablet_service.cc:1467] Processing CreateTablet for tablet 387e9ef5899541b0be27a41dbcf4475c (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=05e7c365fc854833a30f18097c117d10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:56.439100 30139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387e9ef5899541b0be27a41dbcf4475c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:56.442370 30140 tablet_service.cc:1467] Processing CreateTablet for tablet ec6c977cac054b758a43d846e821707a (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=05e7c365fc854833a30f18097c117d10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:56.442763 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6c977cac054b758a43d846e821707a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:56.444170 30138 tablet_service.cc:1467] Processing CreateTablet for tablet 201d79895c2741dd9d9de0cc9f605a91 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=05e7c365fc854833a30f18097c117d10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:56.444617 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 201d79895c2741dd9d9de0cc9f605a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:56.447733 30401 tablet_service.cc:1467] Processing CreateTablet for tablet 201d79895c2741dd9d9de0cc9f605a91 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=05e7c365fc854833a30f18097c117d10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:56.447918 30268 tablet_service.cc:1467] Processing CreateTablet for tablet ec6c977cac054b758a43d846e821707a (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=05e7c365fc854833a30f18097c117d10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:56.448177 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 201d79895c2741dd9d9de0cc9f605a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:56.448431 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6c977cac054b758a43d846e821707a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:56.451681 30984 tablet_bootstrap.cc:492] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:56.454454 30958 tablet_bootstrap.cc:492] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:17:56.454658 30984 tablet_bootstrap.cc:654] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:56.455862 30958 tablet_bootstrap.cc:654] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:56.459769 30958 tablet_bootstrap.cc:492] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:17:56.459998 30958 ts_tablet_manager.cc:1367] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:17:56.460932 30958 raft_consensus.cc:348] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:56.461328 30958 raft_consensus.cc:374] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:56.461465 30958 raft_consensus.cc:729] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:17:56.461920 30958 consensus_queue.cc:260] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:56.466595 30958 ts_tablet_manager.cc:1398] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:17:56.478133 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:17:56.480916 30956 tablet_bootstrap.cc:492] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:56.482558 30956 tablet_bootstrap.cc:654] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:56.486055 30956 tablet_bootstrap.cc:492] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:17:56.486217 30956 ts_tablet_manager.cc:1367] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 08:17:56.486985 30956 raft_consensus.cc:348] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:56.487212 30956 raft_consensus.cc:374] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:56.487288 30956 raft_consensus.cc:729] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:17:56.489339 30956 consensus_queue.cc:260] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:56.491001 30290 raft_consensus.cc:1261] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:56.491186 30956 ts_tablet_manager.cc:1398] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:17:56.491719 30874 consensus_queue.cc:1035] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:56.495258 30158 raft_consensus.cc:1261] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:56.496398 30874 consensus_queue.cc:1035] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:56.496497 30956 tablet_bootstrap.cc:492] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:56.498056 30956 tablet_bootstrap.cc:654] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:56.499130 30960 tablet_bootstrap.cc:492] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:56.499384 30989 tablet_bootstrap.cc:492] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:56.500656 30960 tablet_bootstrap.cc:654] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:56.504467 30989 tablet_bootstrap.cc:654] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:56.505007 30985 tablet_bootstrap.cc:492] T 387e9ef5899541b0be27a41dbcf4475c P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:56.506834 27110 catalog_manager.cc:5576] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899) does not have the latest schema for tablet f2d13299ba914a348fbcabef218c57eb (table rebalancer_dynamic_table [id=319468b7ec92492795908f84cc07ff24]). Expected version 1 got 0
I20240502 08:17:56.514163 30956 tablet_bootstrap.cc:492] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:17:56.514420 30956 ts_tablet_manager.cc:1367] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.005s
I20240502 08:17:56.514454 30985 tablet_bootstrap.cc:654] T 387e9ef5899541b0be27a41dbcf4475c P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:56.515190 30956 raft_consensus.cc:348] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:56.515524 30956 raft_consensus.cc:374] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:56.515681 30956 raft_consensus.cc:729] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:17:56.515939 30956 consensus_queue.cc:260] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:56.516525 30956 ts_tablet_manager.cc:1398] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:56.518424 30956 tablet_bootstrap.cc:492] T 201d79895c2741dd9d9de0cc9f605a91 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:56.520181 30956 tablet_bootstrap.cc:654] T 201d79895c2741dd9d9de0cc9f605a91 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:56.525452 30995 tablet.cc:1722] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:56.526693 30960 tablet_bootstrap.cc:492] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:56.526966 30960 ts_tablet_manager.cc:1367] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.004s
I20240502 08:17:56.527319 27110 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet f2d13299ba914a348fbcabef218c57eb (table rebalancer_dynamic_table [id=319468b7ec92492795908f84cc07ff24]). Expected version 1 got 0
I20240502 08:17:56.528218 30960 raft_consensus.cc:348] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:56.528515 30960 raft_consensus.cc:374] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:56.528627 30960 raft_consensus.cc:729] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:56.529210 30960 consensus_queue.cc:260] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:56.529850 30960 ts_tablet_manager.cc:1398] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:56.531922 30995 tablet.cc:1620] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:17:56.533493 30985 tablet_bootstrap.cc:492] T 387e9ef5899541b0be27a41dbcf4475c P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:56.534636 27110 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet f2d13299ba914a348fbcabef218c57eb (table rebalancer_dynamic_table [id=319468b7ec92492795908f84cc07ff24]). Expected version 1 got 0
I20240502 08:17:56.536005 30985 ts_tablet_manager.cc:1367] T 387e9ef5899541b0be27a41dbcf4475c P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.000s
I20240502 08:17:56.536782 30985 raft_consensus.cc:348] T 387e9ef5899541b0be27a41dbcf4475c P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:56.540933 30956 tablet_bootstrap.cc:492] T 201d79895c2741dd9d9de0cc9f605a91 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:17:56.541127 30956 ts_tablet_manager.cc:1367] T 201d79895c2741dd9d9de0cc9f605a91 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.001s
I20240502 08:17:56.541882 30956 raft_consensus.cc:348] T 201d79895c2741dd9d9de0cc9f605a91 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:56.542273 30956 raft_consensus.cc:374] T 201d79895c2741dd9d9de0cc9f605a91 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:56.542371 30956 raft_consensus.cc:729] T 201d79895c2741dd9d9de0cc9f605a91 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:17:56.542676 30956 consensus_queue.cc:260] T 201d79895c2741dd9d9de0cc9f605a91 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:56.544064 30956 ts_tablet_manager.cc:1398] T 201d79895c2741dd9d9de0cc9f605a91 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:56.544334 30985 raft_consensus.cc:374] T 387e9ef5899541b0be27a41dbcf4475c P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:56.544477 30985 raft_consensus.cc:729] T 387e9ef5899541b0be27a41dbcf4475c P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:56.544823 30985 consensus_queue.cc:260] T 387e9ef5899541b0be27a41dbcf4475c P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:56.545814 30982 mvcc.cc:205] Tried to move back new op lower bound from 7023156742092439552 to 7023156741077565440. Current Snapshot: MvccSnapshot[applied={T|T < 7023156742092439552}]
I20240502 08:17:56.551088 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=319468b7ec92492795908f84cc07ff24] alter complete (version 1)
I20240502 08:17:56.551626 30996 tablet.cc:1722] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:56.554733 30984 tablet_bootstrap.cc:492] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:56.554778 27116 catalog_manager.cc:5576] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899) does not have the latest schema for tablet f2d13299ba914a348fbcabef218c57eb (table rebalancer_dynamic_table [id=319468b7ec92492795908f84cc07ff24]). Expected version 1 got 0
I20240502 08:17:56.552311 30996 tablet.cc:1620] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:17:56.555321 30984 ts_tablet_manager.cc:1367] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.104s user 0.011s sys 0.012s
I20240502 08:17:56.555524 30989 tablet_bootstrap.cc:492] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:56.555672 30989 ts_tablet_manager.cc:1367] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.018s
I20240502 08:17:56.556468 30989 raft_consensus.cc:348] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:56.556679 30989 raft_consensus.cc:374] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:56.556771 30989 raft_consensus.cc:729] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:56.557018 30989 consensus_queue.cc:260] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:56.557574 30989 ts_tablet_manager.cc:1398] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:56.556496 30984 raft_consensus.cc:348] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:56.558146 30984 raft_consensus.cc:374] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:56.558313 30984 raft_consensus.cc:729] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:56.561187 30984 consensus_queue.cc:260] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:56.562080 30984 ts_tablet_manager.cc:1398] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:17:56.562903 30984 tablet_bootstrap.cc:492] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:56.565919 30985 ts_tablet_manager.cc:1398] T 387e9ef5899541b0be27a41dbcf4475c P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20240502 08:17:56.568116 30984 tablet_bootstrap.cc:654] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:56.571242 30999 tablet.cc:1722] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:56.574045 27113 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet f2d13299ba914a348fbcabef218c57eb (table rebalancer_dynamic_table [id=319468b7ec92492795908f84cc07ff24]). Expected version 1 got 0
I20240502 08:17:56.576129 30999 tablet.cc:1620] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:17:56.577015 30027 raft_consensus.cc:1203] T 1662e908146944fd8e74d901e497e35d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:56.577394 30981 raft_consensus.cc:482] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:56.577580 30981 raft_consensus.cc:504] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:56.578790 30981 leader_election.cc:288] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:17:56.579538 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6c977cac054b758a43d846e821707a" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:17:56.579751 30290 raft_consensus.cc:2454] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:17:56.585767 30093 leader_election.cc:302] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:56.586062 30981 raft_consensus.cc:2781] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:56.586174 30981 raft_consensus.cc:482] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:56.586282 30981 raft_consensus.cc:3037] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.590898 30981 raft_consensus.cc:504] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:56.591571 30981 leader_election.cc:288] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:17:56.592476 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6c977cac054b758a43d846e821707a" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:56.592661 27652 raft_consensus.cc:2454] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:17:56.593103 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6c977cac054b758a43d846e821707a" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:17:56.593334 30288 raft_consensus.cc:3037] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.593776 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6c977cac054b758a43d846e821707a" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:56.594049 27652 raft_consensus.cc:3037] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.597654 30288 raft_consensus.cc:2454] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:17:56.597934 27652 raft_consensus.cc:2454] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:17:56.598089 30093 leader_election.cc:302] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:56.598449 30981 raft_consensus.cc:2781] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:56.598665 30981 raft_consensus.cc:686] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:17:56.598984 30981 consensus_queue.cc:237] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:56.607168 30984 tablet_bootstrap.cc:492] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:56.607393 30984 ts_tablet_manager.cc:1367] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.003s
I20240502 08:17:56.608439 30984 raft_consensus.cc:348] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:56.608729 30984 raft_consensus.cc:374] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:56.608808 30984 raft_consensus.cc:729] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:56.609099 30984 consensus_queue.cc:260] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:56.609926 30984 ts_tablet_manager.cc:1398] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 08:17:56.621989 27110 catalog_manager.cc:5484] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:56.624547 30953 tablet_bootstrap.cc:492] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:56.635871 27256 consensus_queue.cc:237] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:17:56.639027 30288 raft_consensus.cc:1261] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:56.639294 30160 raft_consensus.cc:1261] T 81e73229c0d145e7bb81761bf4b820dc P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:56.640110 30871 consensus_queue.cc:1035] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:17:56.640650 30871 consensus_queue.cc:1035] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:17:56.641757 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81e73229c0d145e7bb81761bf4b820dc. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:56.649495 30871 raft_consensus.cc:2932] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:17:56.650264 30953 tablet_bootstrap.cc:492] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6: Bootstrap complete.
I20240502 08:17:56.650620 30953 ts_tablet_manager.cc:1367] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.277s user 0.065s sys 0.000s
I20240502 08:17:56.650367 30288 raft_consensus.cc:2932] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:17:56.651448 30953 raft_consensus.cc:348] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:17:56.651073 30160 raft_consensus.cc:2932] T 81e73229c0d145e7bb81761bf4b820dc P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:17:56.651691 30953 raft_consensus.cc:729] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: LEARNER
I20240502 08:17:56.651890 30953 consensus_queue.cc:260] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:17:56.652530 30953 ts_tablet_manager.cc:1398] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240502 08:17:56.653703 27567 tablet.cc:2369] T 387e9ef5899541b0be27a41dbcf4475c P b01e7c8a302d41078fa39cfef665ff59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:56.658759 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 84cda20738d2452da166e31fc6da57a6-db55b42c724246da8d9cb23369826279 received from {username='slave'} at 127.20.207.6:40325
I20240502 08:17:56.658926 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 84cda20738d2452da166e31fc6da57a6-db55b42c724246da8d9cb23369826279 on tablet db55b42c724246da8d9cb23369826279 with peer 84cda20738d2452da166e31fc6da57a6
I20240502 08:17:56.659103 27113 catalog_manager.cc:5484] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:17:56.662750 27115 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:17:56.676437 30995 tablet.cc:1722] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:56.676924 30995 tablet.cc:1620] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:17:56.681067 30999 tablet.cc:1722] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:56.681509 30999 tablet.cc:1620] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:17:56.690052 27114 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=319468b7ec92492795908f84cc07ff24] alter complete (version 2)
I20240502 08:17:56.692993 30996 tablet.cc:1722] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:56.693423 30996 tablet.cc:1620] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:17:56.709828 30288 consensus_queue.cc:237] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:17:56.712757 30422 raft_consensus.cc:1261] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:56.713466 30653 consensus_queue.cc:1035] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:17:56.717417 30653 raft_consensus.cc:2932] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:17:56.718046 30422 raft_consensus.cc:2932] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:17:56.721619 27256 raft_consensus.cc:1261] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:56.722751 27114 catalog_manager.cc:5484] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:56.723704 31021 consensus_queue.cc:1035] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:17:56.740020 30226 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 -> Peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Couldn't send request to peer bf8ac71b15f740a0a928911360773442. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08245cda17c44a77bcde2f864d2045ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:56.742178 27258 raft_consensus.cc:2932] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:17:56.748003 30968 raft_consensus.cc:482] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:56.748162 30968 raft_consensus.cc:504] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:56.748833 27114 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 08:17:56.748955 30968 leader_election.cc:288] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:56.749042 27114 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 08:17:56.750077 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387e9ef5899541b0be27a41dbcf4475c" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:17:56.750293 30160 raft_consensus.cc:2454] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:17:56.753435 27114 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 7bc33dff3dee4b3fbd0620b5d2faaea1
I20240502 08:17:56.754689 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc33dff3dee4b3fbd0620b5d2faaea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:56.754973 27368 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc33dff3dee4b3fbd0620b5d2faaea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:56.755024 30524 tablet_replica.cc:330] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:56.755299 30524 raft_consensus.cc:2229] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:56.755714 30524 raft_consensus.cc:2258] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.756451 30524 ts_tablet_manager.cc:1875] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.757336 29964 leader_election.cc:302] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:56.757586 30968 raft_consensus.cc:2781] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:56.757678 30968 raft_consensus.cc:482] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:56.757763 30968 raft_consensus.cc:3037] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.758162 30523 tablet_replica.cc:330] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:56.758368 30523 raft_consensus.cc:2229] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:56.758621 30523 raft_consensus.cc:2258] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.759291 30523 ts_tablet_manager.cc:1875] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.760308 30968 raft_consensus.cc:504] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:56.760851 30968 leader_election.cc:288] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:56.754004 27114 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet d8c312b65213444f9e946a932d98874c
I20240502 08:17:56.762506 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc33dff3dee4b3fbd0620b5d2faaea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:17:56.762845 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387e9ef5899541b0be27a41dbcf4475c" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:17:56.763062 30160 raft_consensus.cc:3037] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.763530 27368 tablet_service.cc:1514] Processing DeleteTablet for tablet d8c312b65213444f9e946a932d98874c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:56.763763 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet d8c312b65213444f9e946a932d98874c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:56.763773 27114 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 4c7d725414664b64b0ca58babe5df1ff
I20240502 08:17:56.764142 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet d8c312b65213444f9e946a932d98874c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:56.764380 30522 tablet_replica.cc:330] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:56.764565 30522 raft_consensus.cc:2229] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:56.764839 30522 raft_consensus.cc:2258] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.765465 30522 ts_tablet_manager.cc:1875] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.765679 30160 raft_consensus.cc:2454] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:17:56.766227 27499 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c7d725414664b64b0ca58babe5df1ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:17:56.769289 29964 leader_election.cc:302] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:56.769574 30968 raft_consensus.cc:2781] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:56.770390 27368 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c7d725414664b64b0ca58babe5df1ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:56.771469 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c7d725414664b64b0ca58babe5df1ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:56.773056 30522 ts_tablet_manager.cc:1888] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:56.773245 30522 log.cc:1196] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/d8c312b65213444f9e946a932d98874c
I20240502 08:17:56.773733 30522 ts_tablet_manager.cc:1909] T d8c312b65213444f9e946a932d98874c P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:56.774710 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet d8c312b65213444f9e946a932d98874c (table rebalance_test_table_16 [id=34d6118f78a74939856a95191f42c39d]) successfully deleted
I20240502 08:17:56.774788 30522 tablet_replica.cc:330] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:56.775089 30522 raft_consensus.cc:2229] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:56.776508 30523 ts_tablet_manager.cc:1888] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:56.776662 30523 log.cc:1196] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/7bc33dff3dee4b3fbd0620b5d2faaea1
I20240502 08:17:56.776854 30522 raft_consensus.cc:2258] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.777145 30523 ts_tablet_manager.cc:1909] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:56.777637 30522 ts_tablet_manager.cc:1875] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.778324 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 7bc33dff3dee4b3fbd0620b5d2faaea1 (table rebalance_test_table_16 [id=34d6118f78a74939856a95191f42c39d]) successfully deleted
I20240502 08:17:56.769614 30524 ts_tablet_manager.cc:1888] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:56.778751 30524 log.cc:1196] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/7bc33dff3dee4b3fbd0620b5d2faaea1
I20240502 08:17:56.779366 30524 ts_tablet_manager.cc:1909] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:56.781369 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet 7bc33dff3dee4b3fbd0620b5d2faaea1 (table rebalance_test_table_16 [id=34d6118f78a74939856a95191f42c39d]) successfully deleted
I20240502 08:17:56.781980 30523 tablet_replica.cc:330] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:56.782572 30523 raft_consensus.cc:2229] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:56.783548 31033 tablet_replica.cc:330] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:17:56.783602 30523 raft_consensus.cc:2258] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.783977 31033 raft_consensus.cc:2229] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:56.784302 30523 ts_tablet_manager.cc:1875] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.784965 30524 tablet_replica.cc:330] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:56.785184 30524 raft_consensus.cc:2229] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:56.785382 30524 raft_consensus.cc:2258] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.785615 31033 raft_consensus.cc:2258] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.787272 30524 ts_tablet_manager.cc:1875] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.787420 27258 consensus_queue.cc:237] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:17:56.788688 30522 ts_tablet_manager.cc:1888] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:56.788894 30522 log.cc:1196] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/4c7d725414664b64b0ca58babe5df1ff
I20240502 08:17:56.789386 30522 ts_tablet_manager.cc:1909] T 4c7d725414664b64b0ca58babe5df1ff P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:56.790163 30968 raft_consensus.cc:686] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:17:56.790329 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 4c7d725414664b64b0ca58babe5df1ff (table rebalance_test_table_16 [id=34d6118f78a74939856a95191f42c39d]) successfully deleted
I20240502 08:17:56.790488 30968 consensus_queue.cc:237] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:56.791342 31033 ts_tablet_manager.cc:1875] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.792659 27114 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:17:56.803923 30995 tablet.cc:1722] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:56.810814 30996 tablet.cc:1722] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:56.811079 27652 raft_consensus.cc:1261] T bd640e48563f458cb2d56c94ed1e0347 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:56.811321 30999 tablet.cc:1722] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:56.812220 27521 raft_consensus.cc:1261] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:56.812532 31009 consensus_queue.cc:1035] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:56.816396 30523 ts_tablet_manager.cc:1888] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:56.816532 30523 log.cc:1196] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/d8c312b65213444f9e946a932d98874c
I20240502 08:17:56.816903 30523 ts_tablet_manager.cc:1909] T d8c312b65213444f9e946a932d98874c P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:56.817780 30523 tablet_replica.cc:330] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:56.817988 30523 raft_consensus.cc:2229] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:56.818250 30523 raft_consensus.cc:2258] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.818917 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet d8c312b65213444f9e946a932d98874c (table rebalance_test_table_16 [id=34d6118f78a74939856a95191f42c39d]) successfully deleted
I20240502 08:17:56.820923 31009 consensus_queue.cc:1035] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:56.821059 30523 ts_tablet_manager.cc:1875] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.822999 30524 ts_tablet_manager.cc:1888] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:56.823148 30524 log.cc:1196] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/d8c312b65213444f9e946a932d98874c
I20240502 08:17:56.823582 30524 ts_tablet_manager.cc:1909] T d8c312b65213444f9e946a932d98874c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:56.824265 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387e9ef5899541b0be27a41dbcf4475c" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:56.824273 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387e9ef5899541b0be27a41dbcf4475c" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:56.824508 27520 raft_consensus.cc:3037] T 387e9ef5899541b0be27a41dbcf4475c P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.824522 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet d8c312b65213444f9e946a932d98874c (table rebalance_test_table_16 [id=34d6118f78a74939856a95191f42c39d]) successfully deleted
I20240502 08:17:56.827286 27520 raft_consensus.cc:2454] T 387e9ef5899541b0be27a41dbcf4475c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:17:56.829875 31033 ts_tablet_manager.cc:1888] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:56.830646 31033 log.cc:1196] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/7bc33dff3dee4b3fbd0620b5d2faaea1
W20240502 08:17:56.830835 27194 consensus_peers.cc:485] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Couldn't send request to peer 84cda20738d2452da166e31fc6da57a6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd640e48563f458cb2d56c94ed1e0347. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:56.831082 27114 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=319468b7ec92492795908f84cc07ff24] alter complete (version 3)
I20240502 08:17:56.831422 30523 ts_tablet_manager.cc:1888] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:56.831534 30523 log.cc:1196] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/4c7d725414664b64b0ca58babe5df1ff
I20240502 08:17:56.831480 31033 ts_tablet_manager.cc:1909] T 7bc33dff3dee4b3fbd0620b5d2faaea1 P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:17:56.831959 30523 ts_tablet_manager.cc:1909] T 4c7d725414664b64b0ca58babe5df1ff P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:56.832566 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 7bc33dff3dee4b3fbd0620b5d2faaea1 (table rebalance_test_table_16 [id=34d6118f78a74939856a95191f42c39d]) successfully deleted
I20240502 08:17:56.833011 31033 tablet_replica.cc:330] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:17:56.833227 31033 raft_consensus.cc:2229] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:56.833490 31033 raft_consensus.cc:2258] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.833490 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 4c7d725414664b64b0ca58babe5df1ff (table rebalance_test_table_16 [id=34d6118f78a74939856a95191f42c39d]) successfully deleted
I20240502 08:17:56.833341 29863 raft_consensus.cc:2932] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:17:56.834483 27521 raft_consensus.cc:2932] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:17:56.841046 27113 catalog_manager.cc:5484] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:56.842288 30874 raft_consensus.cc:482] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:56.842420 30874 raft_consensus.cc:504] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:56.843021 30874 leader_election.cc:288] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:56.843812 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "201d79895c2741dd9d9de0cc9f605a91" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:56.844005 27650 raft_consensus.cc:2454] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:17:56.844494 30358 leader_election.cc:302] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:56.844997 30874 raft_consensus.cc:2781] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:56.844929 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "201d79895c2741dd9d9de0cc9f605a91" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:17:56.845099 30874 raft_consensus.cc:482] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:56.845196 30874 raft_consensus.cc:3037] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.845191 30160 raft_consensus.cc:2454] T 201d79895c2741dd9d9de0cc9f605a91 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:17:56.848481 30874 raft_consensus.cc:504] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:56.849115 30874 leader_election.cc:288] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:56.850126 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "201d79895c2741dd9d9de0cc9f605a91" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:56.850287 27650 raft_consensus.cc:3037] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.850327 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "201d79895c2741dd9d9de0cc9f605a91" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:17:56.850489 30160 raft_consensus.cc:3037] T 201d79895c2741dd9d9de0cc9f605a91 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.852905 27650 raft_consensus.cc:2454] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:17:56.853307 30358 leader_election.cc:302] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:56.853631 30874 raft_consensus.cc:2781] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:56.854172 30160 raft_consensus.cc:2454] T 201d79895c2741dd9d9de0cc9f605a91 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:17:56.854611 30874 raft_consensus.cc:686] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:17:56.854856 30874 consensus_queue.cc:237] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:56.855630 31033 ts_tablet_manager.cc:1875] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.857245 27113 catalog_manager.cc:5484] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:56.859920 27116 catalog_manager.cc:5484] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:56.864674 31033 ts_tablet_manager.cc:1888] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:56.864859 31033 log.cc:1196] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/4c7d725414664b64b0ca58babe5df1ff
I20240502 08:17:56.865319 31033 ts_tablet_manager.cc:1909] T 4c7d725414664b64b0ca58babe5df1ff P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:17:56.866490 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 4c7d725414664b64b0ca58babe5df1ff (table rebalance_test_table_16 [id=34d6118f78a74939856a95191f42c39d]) successfully deleted
I20240502 08:17:56.865835 27652 raft_consensus.cc:2932] T bd640e48563f458cb2d56c94ed1e0347 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:17:56.868343 30160 raft_consensus.cc:1203] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:56.877887 27521 consensus_queue.cc:237] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:17:56.880084 27652 raft_consensus.cc:1261] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:56.880232 27258 raft_consensus.cc:1261] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:17:56.880440 27453 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2776f857a5f47e9b8022639dc54757d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:56.880697 30335 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:56.880990 30883 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:56.883596 30556 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:17:56.884248 27652 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:17:56.884215 27258 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:17:56.889010 27113 catalog_manager.cc:5484] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:17:56.904111 30288 consensus_queue.cc:237] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:17:56.906423 30027 raft_consensus.cc:1261] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:56.906451 30422 raft_consensus.cc:1261] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:56.906960 31021 consensus_queue.cc:1035] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:17:56.907552 30653 consensus_queue.cc:1035] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:17:56.909952 30653 raft_consensus.cc:2932] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:17:56.911038 30027 raft_consensus.cc:2932] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:17:56.911624 30422 raft_consensus.cc:2932] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
W20240502 08:17:56.915444 30225 consensus_peers.cc:485] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd644c6742854b79a212d858fa9df147. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:56.917344 27116 catalog_manager.cc:5484] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:17:56.938793 27113 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:17:56.939018 27113 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:17:56.942195 27113 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet f2d13299ba914a348fbcabef218c57eb
I20240502 08:17:56.943269 30401 tablet_service.cc:1514] Processing DeleteTablet for tablet f2d13299ba914a348fbcabef218c57eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:56694
I20240502 08:17:56.943444 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet f2d13299ba914a348fbcabef218c57eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:52730
I20240502 08:17:56.943991 31067 tablet_replica.cc:330] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:17:56.944100 30139 tablet_service.cc:1514] Processing DeleteTablet for tablet f2d13299ba914a348fbcabef218c57eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:56 UTC) from {username='slave'} at 127.0.0.1:48808
I20240502 08:17:56.945978 31067 raft_consensus.cc:2229] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:56.946528 31067 raft_consensus.cc:2258] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.950603 31069 tablet_replica.cc:330] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:17:56.950871 31069 raft_consensus.cc:2229] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:56.951125 31067 ts_tablet_manager.cc:1875] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.951143 31069 raft_consensus.cc:2258] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.952044 31069 ts_tablet_manager.cc:1875] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.955844 31068 tablet_replica.cc:330] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:17:56.956235 31068 raft_consensus.cc:2229] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:56.956513 31068 raft_consensus.cc:2258] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:56.958112 31068 ts_tablet_manager.cc:1875] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:56.958392 31067 ts_tablet_manager.cc:1888] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:56.958508 31067 log.cc:1196] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/f2d13299ba914a348fbcabef218c57eb
I20240502 08:17:56.958986 31067 ts_tablet_manager.cc:1909] T f2d13299ba914a348fbcabef218c57eb P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting consensus metadata
I20240502 08:17:56.960870 31069 ts_tablet_manager.cc:1888] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:56.961045 31069 log.cc:1196] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/f2d13299ba914a348fbcabef218c57eb
I20240502 08:17:56.961630 31069 ts_tablet_manager.cc:1909] T f2d13299ba914a348fbcabef218c57eb P 84cda20738d2452da166e31fc6da57a6: Deleting consensus metadata
I20240502 08:17:56.961658 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet f2d13299ba914a348fbcabef218c57eb (table rebalancer_dynamic_table_new_name [id=319468b7ec92492795908f84cc07ff24]) successfully deleted
I20240502 08:17:56.962709 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet f2d13299ba914a348fbcabef218c57eb (table rebalancer_dynamic_table_new_name [id=319468b7ec92492795908f84cc07ff24]) successfully deleted
I20240502 08:17:56.963872 31068 ts_tablet_manager.cc:1888] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:56.964016 31068 log.cc:1196] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/f2d13299ba914a348fbcabef218c57eb
I20240502 08:17:56.964493 31068 ts_tablet_manager.cc:1909] T f2d13299ba914a348fbcabef218c57eb P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting consensus metadata
I20240502 08:17:56.966063 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet f2d13299ba914a348fbcabef218c57eb (table rebalancer_dynamic_table_new_name [id=319468b7ec92492795908f84cc07ff24]) successfully deleted
I20240502 08:17:57.046095 27110 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:17:57.058636 30268 tablet_service.cc:1467] Processing CreateTablet for tablet e8a610177b03466eb454c731ac756f1e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=885b6edfa1754349a3060bb856f6f829]), partition=
I20240502 08:17:57.059387 27368 tablet_service.cc:1467] Processing CreateTablet for tablet e8a610177b03466eb454c731ac756f1e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=885b6edfa1754349a3060bb856f6f829]), partition=
I20240502 08:17:57.059484 30401 tablet_service.cc:1467] Processing CreateTablet for tablet e8a610177b03466eb454c731ac756f1e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=885b6edfa1754349a3060bb856f6f829]), partition=
I20240502 08:17:57.059777 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a610177b03466eb454c731ac756f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.059827 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a610177b03466eb454c731ac756f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.066051 31093 tablet_bootstrap.cc:492] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:17:57.066077 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a610177b03466eb454c731ac756f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.067677 31093 tablet_bootstrap.cc:654] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.069041 31092 tablet_bootstrap.cc:492] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:57.072520 31092 tablet_bootstrap.cc:654] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.074234 31092 tablet_bootstrap.cc:492] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:57.074422 31092 ts_tablet_manager.cc:1367] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240502 08:17:57.075059 31092 raft_consensus.cc:348] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:57.075219 31092 raft_consensus.cc:374] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.075316 31092 raft_consensus.cc:729] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.075526 31092 consensus_queue.cc:260] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:57.076102 31092 ts_tablet_manager.cc:1398] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:57.096132 31096 tablet_bootstrap.cc:492] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:57.097847 31096 tablet_bootstrap.cc:654] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.101313 31093 tablet_bootstrap.cc:492] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:17:57.101557 31093 ts_tablet_manager.cc:1367] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.014s
I20240502 08:17:57.102460 31093 raft_consensus.cc:348] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:57.102772 31093 raft_consensus.cc:374] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.102923 31093 raft_consensus.cc:729] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.103111 31096 tablet_bootstrap.cc:492] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:57.103242 31096 ts_tablet_manager.cc:1367] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 08:17:57.103349 31093 consensus_queue.cc:260] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:57.103890 31096 raft_consensus.cc:348] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:57.104106 31096 raft_consensus.cc:374] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.104413 31096 raft_consensus.cc:729] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.104650 31096 consensus_queue.cc:260] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:57.105172 31096 ts_tablet_manager.cc:1398] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:57.106501 31093 ts_tablet_manager.cc:1398] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:17:57.107601 31050 raft_consensus.cc:482] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:57.107765 31050 raft_consensus.cc:504] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:57.108721 31050 leader_election.cc:288] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:17:57.109652 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a610177b03466eb454c731ac756f1e" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:17:57.109946 30422 raft_consensus.cc:2454] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:57.110960 27323 leader_election.cc:302] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:57.111538 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a610177b03466eb454c731ac756f1e" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:17:57.111799 30288 raft_consensus.cc:2454] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:57.112550 31050 raft_consensus.cc:2781] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:57.112694 31050 raft_consensus.cc:482] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:57.112800 31050 raft_consensus.cc:3037] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.116566 31050 raft_consensus.cc:504] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:57.118682 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a610177b03466eb454c731ac756f1e" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:17:57.118882 30288 raft_consensus.cc:3037] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.119377 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a610177b03466eb454c731ac756f1e" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:17:57.119648 30422 raft_consensus.cc:3037] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.122859 30422 raft_consensus.cc:2454] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:57.123387 27323 leader_election.cc:302] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:57.123684 31049 raft_consensus.cc:2781] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:57.123803 31049 raft_consensus.cc:686] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:17:57.124123 31049 consensus_queue.cc:237] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:57.126940 31050 leader_election.cc:288] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:17:57.128384 30288 raft_consensus.cc:2454] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:57.132663 27238 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:17:57.145665 27113 catalog_manager.cc:5484] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:57.157558 30981 consensus_queue.cc:1035] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:57.163748 30871 raft_consensus.cc:1050] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 to VOTER
I20240502 08:17:57.165055 30871 consensus_queue.cc:237] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:17:57.178632 30401 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:57.180452 30008 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:57.181334 27390 raft_consensus.cc:1261] T 1662e908146944fd8e74d901e497e35d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:57.181993 29863 consensus_queue.cc:1035] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:57.190421 27521 raft_consensus.cc:1261] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:57.192266 29863 consensus_queue.cc:1035] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:57.193146 30027 raft_consensus.cc:1261] T 1662e908146944fd8e74d901e497e35d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:57.193630 30871 consensus_queue.cc:1035] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:57.202890 30268 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:57.220358 31009 raft_consensus.cc:2932] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:17:57.221158 27520 raft_consensus.cc:2932] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:17:57.227928 31113 consensus_queue.cc:1035] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:57.228083 27113 catalog_manager.cc:5484] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:17:57.234544 31115 ts_tablet_manager.cc:902] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442: Initiating tablet copy from peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:17:57.243932 31117 ts_tablet_manager.cc:902] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:57.261674 31115 tablet_copy_client.cc:308] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.7:40343
I20240502 08:17:57.261387 30027 raft_consensus.cc:2932] T 1662e908146944fd8e74d901e497e35d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:17:57.263090 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:17:57.268371 30968 consensus_queue.cc:1035] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:57.275566 27110 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 08:17:57.275768 27110 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 08:17:57.284019 27110 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet f733b0e3238f498cb54e3b2beef24249
I20240502 08:17:57.284729 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet f733b0e3238f498cb54e3b2beef24249 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:57 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:57.285019 30522 tablet_replica.cc:330] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:57.285244 30522 raft_consensus.cc:2229] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:57.285450 30522 raft_consensus.cc:2258] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:57.286192 30522 ts_tablet_manager.cc:1875] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:57.286237 27110 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet a67cb9c98522464399a44186109333cf
I20240502 08:17:57.287493 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet a67cb9c98522464399a44186109333cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:57 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:17:57.287760 31033 tablet_replica.cc:330] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:17:57.287941 31033 raft_consensus.cc:2229] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:57.288163 31033 raft_consensus.cc:2258] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:57.288589 27258 consensus_queue.cc:237] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:17:57.288861 31033 ts_tablet_manager.cc:1875] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:57.290246 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet f733b0e3238f498cb54e3b2beef24249 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:57 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:57.290524 30524 tablet_replica.cc:330] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:57.290730 30524 raft_consensus.cc:2229] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:57.290800 27110 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 549102ad342342f49049f34ebac5bc7d
I20240502 08:17:57.290985 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet a67cb9c98522464399a44186109333cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:57 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:57.291067 30524 raft_consensus.cc:2258] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:57.291393 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet 549102ad342342f49049f34ebac5bc7d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:57 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:17:57.291889 30524 ts_tablet_manager.cc:1875] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:57.291918 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet 549102ad342342f49049f34ebac5bc7d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:57 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:57.294023 30522 ts_tablet_manager.cc:1888] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:57.294202 30522 log.cc:1196] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/f733b0e3238f498cb54e3b2beef24249
I20240502 08:17:57.294716 30522 ts_tablet_manager.cc:1909] T f733b0e3238f498cb54e3b2beef24249 P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:57.295756 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet f733b0e3238f498cb54e3b2beef24249 (table rebalance_test_table_18 [id=9df8e288a677400882b593a841c5c051]) successfully deleted
I20240502 08:17:57.295810 30522 tablet_replica.cc:330] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:57.296269 30522 raft_consensus.cc:2229] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:57.297317 30522 raft_consensus.cc:2258] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:57.299149 30524 ts_tablet_manager.cc:1888] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:57.299304 30524 log.cc:1196] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/f733b0e3238f498cb54e3b2beef24249
I20240502 08:17:57.299382 31033 ts_tablet_manager.cc:1888] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:57.299510 31033 log.cc:1196] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/a67cb9c98522464399a44186109333cf
I20240502 08:17:57.299832 30524 ts_tablet_manager.cc:1909] T f733b0e3238f498cb54e3b2beef24249 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:57.300012 31033 ts_tablet_manager.cc:1909] T a67cb9c98522464399a44186109333cf P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:17:57.300770 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet f733b0e3238f498cb54e3b2beef24249 (table rebalance_test_table_18 [id=9df8e288a677400882b593a841c5c051]) successfully deleted
I20240502 08:17:57.301105 30524 tablet_replica.cc:330] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:57.301317 30524 raft_consensus.cc:2229] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:57.302104 30524 raft_consensus.cc:2258] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:57.302745 30524 ts_tablet_manager.cc:1875] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:57.304925 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet a67cb9c98522464399a44186109333cf (table rebalance_test_table_18 [id=9df8e288a677400882b593a841c5c051]) successfully deleted
I20240502 08:17:57.310675 30028 raft_consensus.cc:1261] T 1662e908146944fd8e74d901e497e35d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:57.311980 30874 consensus_queue.cc:1035] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:57.316185 30871 consensus_queue.cc:1035] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:57.319551 29863 raft_consensus.cc:2932] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:17:57.320097 27500 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:17:57.320830 30310 tablet_copy_service.cc:140] P 49c5c8f70e494edd9fcda40cc0af6cc4: Received BeginTabletCopySession request for tablet 08245cda17c44a77bcde2f864d2045ec from peer bf8ac71b15f740a0a928911360773442 ({username='slave'} at 127.20.207.4:32967)
I20240502 08:17:57.321067 30310 tablet_copy_service.cc:161] P 49c5c8f70e494edd9fcda40cc0af6cc4: Beginning new tablet copy session on tablet 08245cda17c44a77bcde2f864d2045ec from peer bf8ac71b15f740a0a928911360773442 at {username='slave'} at 127.20.207.4:32967: session id = bf8ac71b15f740a0a928911360773442-08245cda17c44a77bcde2f864d2045ec
I20240502 08:17:57.324072 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1662e908146944fd8e74d901e497e35d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:57.324847 30524 ts_tablet_manager.cc:1888] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:57.324940 30310 tablet_copy_source_session.cc:205] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:57.325012 30524 log.cc:1196] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/549102ad342342f49049f34ebac5bc7d
I20240502 08:17:57.325484 30524 ts_tablet_manager.cc:1909] T 549102ad342342f49049f34ebac5bc7d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:17:57.326593 30028 raft_consensus.cc:2932] T 1662e908146944fd8e74d901e497e35d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:17:57.327239 27113 catalog_manager.cc:5484] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:57.328352 31115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08245cda17c44a77bcde2f864d2045ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.331125 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet 1662e908146944fd8e74d901e497e35d with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52776
I20240502 08:17:57.335628 31033 tablet_replica.cc:330] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:17:57.336272 27370 tablet_service.cc:1514] Processing DeleteTablet for tablet f733b0e3238f498cb54e3b2beef24249 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:57 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:57.336966 31033 raft_consensus.cc:2229] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:57.339975 31033 raft_consensus.cc:2258] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:57.341153 31033 ts_tablet_manager.cc:1875] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:57.343050 27367 tablet_service.cc:1514] Processing DeleteTablet for tablet 549102ad342342f49049f34ebac5bc7d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:57 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:57.345520 27369 tablet_service.cc:1514] Processing DeleteTablet for tablet a67cb9c98522464399a44186109333cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:57 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:57.345782 27366 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:17:57.350296 30139 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:57.359884 27390 raft_consensus.cc:2932] T 1662e908146944fd8e74d901e497e35d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:17:57.365803 31135 consensus_queue.cc:1035] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:57.375367 30422 raft_consensus.cc:1261] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:57.375998 31049 consensus_queue.cc:1035] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:57.377947 30874 consensus_queue.cc:1035] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:57.380147 30288 raft_consensus.cc:1261] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:57.380249 27113 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet e8a610177b03466eb454c731ac756f1e (table rebalancer_dynamic_table [id=885b6edfa1754349a3060bb856f6f829]). Expected version 1 got 0
I20240502 08:17:57.381841 30826 consensus_queue.cc:1035] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:57.384406 27388 raft_consensus.cc:1261] T 1662e908146944fd8e74d901e497e35d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:57.385514 30871 consensus_queue.cc:1035] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:57.387151 27388 raft_consensus.cc:2932] T 1662e908146944fd8e74d901e497e35d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:17:57.402094 31147 tablet.cc:1722] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:57.402606 31147 tablet.cc:1620] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:17:57.406281 30522 ts_tablet_manager.cc:1875] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:57.412385 31115 tablet_copy_client.cc:783] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:57.412894 31115 tablet_copy_client.cc:647] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:57.415921 31140 ts_tablet_manager.cc:902] T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1: Initiating tablet copy from peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:17:57.418998 31115 tablet_copy_client.cc:523] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:57.419888 27631 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:57.429297 31137 tablet_replica.cc:330] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:57.429834 31137 raft_consensus.cc:2229] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:57.430622 31137 raft_consensus.cc:2258] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:57.431205 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet 549102ad342342f49049f34ebac5bc7d (table rebalance_test_table_18 [id=9df8e288a677400882b593a841c5c051]) successfully deleted
I20240502 08:17:57.431576 31137 ts_tablet_manager.cc:1875] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:57.434439 27113 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:57.435505 31033 ts_tablet_manager.cc:1888] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:17:57.435649 31033 log.cc:1196] T 1662e908146944fd8e74d901e497e35d P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/1662e908146944fd8e74d901e497e35d
I20240502 08:17:57.436319 31131 ts_tablet_manager.cc:902] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:57.438437 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 1662e908146944fd8e74d901e497e35d (table rebalance_test_table_15 [id=f2de4023ae664751b94f83ff4434cd92]) successfully deleted
I20240502 08:17:57.444391 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 26545a868bd6415c9c9db4869b2841c7 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9df4632350e6444089d25e0f410be77a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:57.444864 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26545a868bd6415c9c9db4869b2841c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.445624 30008 tablet_service.cc:1467] Processing CreateTablet for tablet 26545a868bd6415c9c9db4869b2841c7 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9df4632350e6444089d25e0f410be77a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:57.446031 30008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26545a868bd6415c9c9db4869b2841c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.449187 30006 tablet_service.cc:1467] Processing CreateTablet for tablet 84953bd8daba4885be1f5f9ad4149aba (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9df4632350e6444089d25e0f410be77a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:57.449388 30401 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5a0175799e478c9f53ecc82ba090ae (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9df4632350e6444089d25e0f410be77a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:57.449530 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84953bd8daba4885be1f5f9ad4149aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.449765 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5a0175799e478c9f53ecc82ba090ae. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:57.449936 30358 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.6:38899, user_credentials={real_user=slave}} blocked reactor thread for 72240.4us
I20240502 08:17:57.458043 30270 tablet_service.cc:1467] Processing CreateTablet for tablet 84953bd8daba4885be1f5f9ad4149aba (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9df4632350e6444089d25e0f410be77a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:57.458535 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84953bd8daba4885be1f5f9ad4149aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.461401 30269 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5a0175799e478c9f53ecc82ba090ae (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9df4632350e6444089d25e0f410be77a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:57.461781 30269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5a0175799e478c9f53ecc82ba090ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.462492 31048 raft_consensus.cc:1050] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 84cda20738d2452da166e31fc6da57a6 to VOTER
I20240502 08:17:57.463482 31048 consensus_queue.cc:237] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:17:57.466720 27258 raft_consensus.cc:1261] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:57.467300 30556 consensus_queue.cc:1035] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:57.470091 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 84953bd8daba4885be1f5f9ad4149aba (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9df4632350e6444089d25e0f410be77a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:57.470530 31154 tablet.cc:1722] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:57.470654 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84953bd8daba4885be1f5f9ad4149aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.470683 30139 tablet_service.cc:1467] Processing CreateTablet for tablet 26545a868bd6415c9c9db4869b2841c7 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9df4632350e6444089d25e0f410be77a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:57.471112 30139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26545a868bd6415c9c9db4869b2841c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.474341 31154 tablet.cc:1620] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:17:57.474341 30138 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5a0175799e478c9f53ecc82ba090ae (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=9df4632350e6444089d25e0f410be77a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:57.474676 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5a0175799e478c9f53ecc82ba090ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.479290 27107 catalog_manager.cc:5576] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291) does not have the latest schema for tablet e8a610177b03466eb454c731ac756f1e (table rebalancer_dynamic_table [id=885b6edfa1754349a3060bb856f6f829]). Expected version 1 got 0
I20240502 08:17:57.482326 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=885b6edfa1754349a3060bb856f6f829] alter complete (version 1)
I20240502 08:17:57.489111 30158 raft_consensus.cc:1261] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:57.489737 30335 consensus_queue.cc:1035] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:57.492022 31161 tablet_bootstrap.cc:492] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:57.496230 31093 tablet_bootstrap.cc:492] T 0a5a0175799e478c9f53ecc82ba090ae P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:17:57.497539 31096 tablet_bootstrap.cc:492] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:57.498054 31093 tablet_bootstrap.cc:654] T 0a5a0175799e478c9f53ecc82ba090ae P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.498095 31156 tablet_bootstrap.cc:492] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:57.499130 31096 tablet_bootstrap.cc:654] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.499732 31156 tablet_bootstrap.cc:654] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.502569 31160 tablet_bootstrap.cc:492] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:57.505120 27107 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet e8a610177b03466eb454c731ac756f1e (table rebalancer_dynamic_table [id=885b6edfa1754349a3060bb856f6f829]). Expected version 1 got 0
I20240502 08:17:57.505682 31115 tablet_bootstrap.cc:492] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:57.515162 31093 tablet_bootstrap.cc:492] T 0a5a0175799e478c9f53ecc82ba090ae P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:17:57.515416 31093 ts_tablet_manager.cc:1367] T 0a5a0175799e478c9f53ecc82ba090ae P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.007s
I20240502 08:17:57.517266 31093 raft_consensus.cc:348] T 0a5a0175799e478c9f53ecc82ba090ae P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:57.517632 31093 raft_consensus.cc:374] T 0a5a0175799e478c9f53ecc82ba090ae P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.517746 31093 raft_consensus.cc:729] T 0a5a0175799e478c9f53ecc82ba090ae P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.518091 31093 consensus_queue.cc:260] T 0a5a0175799e478c9f53ecc82ba090ae P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:57.518741 31093 ts_tablet_manager.cc:1398] T 0a5a0175799e478c9f53ecc82ba090ae P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:57.521261 30522 ts_tablet_manager.cc:1888] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:57.523305 31161 tablet_bootstrap.cc:654] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.524298 31156 tablet_bootstrap.cc:492] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:57.524914 31131 tablet_copy_client.cc:308] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:17:57.437263 31146 tablet.cc:1722] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:57.449278 31117 tablet_copy_client.cc:308] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:17:57.529904 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet 81e73229c0d145e7bb81761bf4b820dc from peer 83e171a5e16c4008923f1d43fc5ce7b1 ({username='slave'} at 127.20.207.2:44351)
I20240502 08:17:57.530092 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet 81e73229c0d145e7bb81761bf4b820dc from peer 83e171a5e16c4008923f1d43fc5ce7b1 at {username='slave'} at 127.20.207.2:44351: session id = 83e171a5e16c4008923f1d43fc5ce7b1-81e73229c0d145e7bb81761bf4b820dc
I20240502 08:17:57.533002 27278 tablet_copy_source_session.cc:205] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:57.535296 31117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81e73229c0d145e7bb81761bf4b820dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.526332 31160 tablet_bootstrap.cc:654] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.538141 30522 log.cc:1196] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/a67cb9c98522464399a44186109333cf
I20240502 08:17:57.471122 27388 raft_consensus.cc:1261] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 08:17:57.481185 27326 consensus_peers.cc:485] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a67cb9c98522464399a44186109333cf. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:57.538682 27326 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.3:44469, user_credentials={real_user=slave}} blocked reactor thread for 57557.7us
I20240502 08:17:57.510290 31140 tablet_copy_client.cc:308] T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.7:40343
I20240502 08:17:57.539774 30310 tablet_copy_service.cc:140] P 49c5c8f70e494edd9fcda40cc0af6cc4: Received BeginTabletCopySession request for tablet dd644c6742854b79a212d858fa9df147 from peer 83e171a5e16c4008923f1d43fc5ce7b1 ({username='slave'} at 127.20.207.2:58817)
I20240502 08:17:57.539963 30310 tablet_copy_service.cc:161] P 49c5c8f70e494edd9fcda40cc0af6cc4: Beginning new tablet copy session on tablet dd644c6742854b79a212d858fa9df147 from peer 83e171a5e16c4008923f1d43fc5ce7b1 at {username='slave'} at 127.20.207.2:58817: session id = 83e171a5e16c4008923f1d43fc5ce7b1-dd644c6742854b79a212d858fa9df147
W20240502 08:17:57.540524 27326 consensus_peers.cc:485] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Couldn't send request to peer bf8ac71b15f740a0a928911360773442. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:57.541399 30335 consensus_queue.cc:1035] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:57.543125 30310 tablet_copy_source_session.cc:205] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:57.544378 31140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd644c6742854b79a212d858fa9df147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.551496 31156 ts_tablet_manager.cc:1367] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.000s
I20240502 08:17:57.551913 31117 tablet_copy_client.cc:783] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:57.552316 31117 tablet_copy_client.cc:647] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:57.552338 31156 raft_consensus.cc:348] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:57.552616 31156 raft_consensus.cc:374] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.552780 31156 raft_consensus.cc:729] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.553122 31156 consensus_queue.cc:260] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:57.553941 31156 ts_tablet_manager.cc:1398] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:57.554270 31156 tablet_bootstrap.cc:492] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:57.556262 31156 tablet_bootstrap.cc:654] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.557067 30522 ts_tablet_manager.cc:1909] T a67cb9c98522464399a44186109333cf P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:57.556563 31160 tablet_bootstrap.cc:492] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:57.557313 31160 ts_tablet_manager.cc:1367] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.055s user 0.006s sys 0.000s
I20240502 08:17:57.557477 30335 raft_consensus.cc:2932] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:17:57.558429 31160 raft_consensus.cc:348] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:57.558962 31160 raft_consensus.cc:374] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.559053 31160 raft_consensus.cc:729] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.559275 31160 consensus_queue.cc:260] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:57.560665 31160 ts_tablet_manager.cc:1398] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:57.564330 31161 tablet_bootstrap.cc:492] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:17:57.564544 31161 ts_tablet_manager.cc:1367] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.073s user 0.007s sys 0.000s
I20240502 08:17:57.565207 31137 ts_tablet_manager.cc:1888] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:57.565347 31137 log.cc:1196] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/f733b0e3238f498cb54e3b2beef24249
I20240502 08:17:57.565063 27258 raft_consensus.cc:2932] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:17:57.565814 31137 ts_tablet_manager.cc:1909] T f733b0e3238f498cb54e3b2beef24249 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:57.565758 31161 raft_consensus.cc:348] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:57.566162 31161 raft_consensus.cc:374] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.566308 31161 raft_consensus.cc:729] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.566794 31137 tablet_replica.cc:330] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:57.566622 31161 consensus_queue.cc:260] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:57.567014 31137 raft_consensus.cc:2229] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:57.567359 31137 raft_consensus.cc:2258] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:57.568046 31137 ts_tablet_manager.cc:1875] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:57.568481 31161 ts_tablet_manager.cc:1398] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:57.568806 31161 tablet_bootstrap.cc:492] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:57.569470 30522 tablet_replica.cc:330] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:17:57.569653 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet f733b0e3238f498cb54e3b2beef24249 (table rebalance_test_table_18 [id=9df8e288a677400882b593a841c5c051]) successfully deleted
I20240502 08:17:57.569738 30522 raft_consensus.cc:2229] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:57.570075 30522 raft_consensus.cc:2258] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:57.570312 27390 raft_consensus.cc:2932] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:17:57.570855 30522 ts_tablet_manager.cc:1875] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:57.571561 31146 tablet.cc:1620] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1: MemRowSet was empty: no flush needed.
I20240502 08:17:57.572855 27112 catalog_manager.cc:5484] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:17:57.574075 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet a67cb9c98522464399a44186109333cf (table rebalance_test_table_18 [id=9df8e288a677400882b593a841c5c051]) successfully deleted
I20240502 08:17:57.578554 31117 tablet_copy_client.cc:523] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:57.590431 31140 tablet_copy_client.cc:783] T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:57.590453 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet bd640e48563f458cb2d56c94ed1e0347 from peer 84cda20738d2452da166e31fc6da57a6 ({username='slave'} at 127.20.207.6:52973)
I20240502 08:17:57.590608 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet bd640e48563f458cb2d56c94ed1e0347 from peer 84cda20738d2452da166e31fc6da57a6 at {username='slave'} at 127.20.207.6:52973: session id = 84cda20738d2452da166e31fc6da57a6-bd640e48563f458cb2d56c94ed1e0347
I20240502 08:17:57.590689 31140 tablet_copy_client.cc:647] T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:57.592721 31161 tablet_bootstrap.cc:654] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.593636 27278 tablet_copy_source_session.cc:205] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:57.593856 31140 tablet_copy_client.cc:523] T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:57.599083 31131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd640e48563f458cb2d56c94ed1e0347. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.607586 31131 tablet_copy_client.cc:783] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:57.607821 31131 tablet_copy_client.cc:647] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:57.610338 31131 tablet_copy_client.cc:523] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:57.618309 31140 tablet_bootstrap.cc:492] T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:57.622956 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:17:57.634521 31131 tablet_bootstrap.cc:492] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:57.640697 31147 tablet.cc:1722] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:57.641131 31147 tablet.cc:1620] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:17:57.642788 31167 ts_tablet_manager.cc:902] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:57.644392 31096 tablet_bootstrap.cc:492] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:57.644661 31096 ts_tablet_manager.cc:1367] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.147s user 0.012s sys 0.000s
I20240502 08:17:57.645684 31096 raft_consensus.cc:348] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:57.645967 31096 raft_consensus.cc:374] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.646095 31096 raft_consensus.cc:729] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.646389 31096 consensus_queue.cc:260] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:57.648062 31096 ts_tablet_manager.cc:1398] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:57.648384 31096 tablet_bootstrap.cc:492] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:57.649557 27113 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=885b6edfa1754349a3060bb856f6f829] alter complete (version 2)
I20240502 08:17:57.650285 31154 tablet.cc:1722] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:57.650857 31154 tablet.cc:1620] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:17:57.653534 31117 tablet_bootstrap.cc:492] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:57.663447 27110 catalog_manager.cc:5576] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291) does not have the latest schema for tablet e8a610177b03466eb454c731ac756f1e (table rebalancer_dynamic_table [id=885b6edfa1754349a3060bb856f6f829]). Expected version 2 got 1
I20240502 08:17:57.670979 27107 catalog_manager.cc:5576] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291) does not have the latest schema for tablet e8a610177b03466eb454c731ac756f1e (table rebalancer_dynamic_table [id=885b6edfa1754349a3060bb856f6f829]). Expected version 2 got 1
I20240502 08:17:57.671092 31140 tablet_bootstrap.cc:492] T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:57.671568 31140 tablet_bootstrap.cc:492] T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap complete.
I20240502 08:17:57.671835 31140 ts_tablet_manager.cc:1367] T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.004s
I20240502 08:17:57.672715 31140 raft_consensus.cc:348] T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:17:57.673025 31140 raft_consensus.cc:729] T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: LEARNER
I20240502 08:17:57.673274 31140 consensus_queue.cc:260] T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:17:57.673920 31140 ts_tablet_manager.cc:1398] T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240502 08:17:57.674741 30310 tablet_copy_service.cc:342] P 49c5c8f70e494edd9fcda40cc0af6cc4: Request end of tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-dd644c6742854b79a212d858fa9df147 received from {username='slave'} at 127.20.207.2:58817
I20240502 08:17:57.674870 30310 tablet_copy_service.cc:433] P 49c5c8f70e494edd9fcda40cc0af6cc4: ending tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-dd644c6742854b79a212d858fa9df147 on tablet dd644c6742854b79a212d858fa9df147 with peer 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:17:57.680086 31115 tablet_bootstrap.cc:492] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:57.683775 31146 tablet.cc:1722] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:57.684566 31146 tablet.cc:1620] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1: MemRowSet was empty: no flush needed.
I20240502 08:17:57.686302 31115 tablet_bootstrap.cc:492] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442: Bootstrap complete.
I20240502 08:17:57.686767 31115 ts_tablet_manager.cc:1367] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.181s user 0.009s sys 0.000s
I20240502 08:17:57.690034 30522 ts_tablet_manager.cc:1888] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:57.693482 31096 tablet_bootstrap.cc:654] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.697031 31115 raft_consensus.cc:348] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:17:57.702109 31115 raft_consensus.cc:729] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: LEARNER
I20240502 08:17:57.702531 31115 consensus_queue.cc:260] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:17:57.703013 31167 tablet_copy_client.cc:308] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:17:57.694227 30522 log.cc:1196] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/549102ad342342f49049f34ebac5bc7d
I20240502 08:17:57.704329 30522 ts_tablet_manager.cc:1909] T 549102ad342342f49049f34ebac5bc7d P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:17:57.710870 31115 ts_tablet_manager.cc:1398] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240502 08:17:57.710953 31161 tablet_bootstrap.cc:492] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:17:57.711709 31161 ts_tablet_manager.cc:1367] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.143s user 0.026s sys 0.028s
I20240502 08:17:57.711763 31131 tablet_bootstrap.cc:492] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:57.712738 31131 tablet_bootstrap.cc:492] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6: Bootstrap complete.
I20240502 08:17:57.713033 31131 ts_tablet_manager.cc:1367] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.099s user 0.049s sys 0.006s
I20240502 08:17:57.713593 31161 raft_consensus.cc:348] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:57.714179 31096 tablet_bootstrap.cc:492] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:57.714344 31096 ts_tablet_manager.cc:1367] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.066s user 0.001s sys 0.002s
I20240502 08:17:57.713893 31131 raft_consensus.cc:348] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:17:57.714375 31161 raft_consensus.cc:374] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.714470 31131 raft_consensus.cc:729] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: LEARNER
I20240502 08:17:57.714677 31161 raft_consensus.cc:729] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.714852 31131 consensus_queue.cc:260] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:17:57.715348 31161 consensus_queue.cc:260] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:57.715857 31096 raft_consensus.cc:348] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:57.716059 31096 raft_consensus.cc:374] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.716166 31096 raft_consensus.cc:729] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.716297 31131 ts_tablet_manager.cc:1398] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:57.716383 31096 consensus_queue.cc:260] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:57.716914 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session 84cda20738d2452da166e31fc6da57a6-bd640e48563f458cb2d56c94ed1e0347 received from {username='slave'} at 127.20.207.6:52973
I20240502 08:17:57.717067 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session 84cda20738d2452da166e31fc6da57a6-bd640e48563f458cb2d56c94ed1e0347 on tablet bd640e48563f458cb2d56c94ed1e0347 with peer 84cda20738d2452da166e31fc6da57a6
I20240502 08:17:57.718761 30160 raft_consensus.cc:2932] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:17:57.722689 31117 tablet_bootstrap.cc:492] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:57.723135 31117 tablet_bootstrap.cc:492] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap complete.
I20240502 08:17:57.723021 27520 consensus_queue.cc:237] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:17:57.723376 31161 ts_tablet_manager.cc:1398] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 08:17:57.724159 30310 tablet_copy_service.cc:342] P 49c5c8f70e494edd9fcda40cc0af6cc4: Request end of tablet copy session bf8ac71b15f740a0a928911360773442-08245cda17c44a77bcde2f864d2045ec received from {username='slave'} at 127.20.207.4:32967
I20240502 08:17:57.724294 30310 tablet_copy_service.cc:433] P 49c5c8f70e494edd9fcda40cc0af6cc4: ending tablet copy session bf8ac71b15f740a0a928911360773442-08245cda17c44a77bcde2f864d2045ec on tablet 08245cda17c44a77bcde2f864d2045ec with peer bf8ac71b15f740a0a928911360773442
I20240502 08:17:57.725314 27390 raft_consensus.cc:1261] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:57.725317 30160 raft_consensus.cc:1261] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:57.725466 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 549102ad342342f49049f34ebac5bc7d (table rebalance_test_table_18 [id=9df8e288a677400882b593a841c5c051]) successfully deleted
I20240502 08:17:57.725852 30556 consensus_queue.cc:1035] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:57.726333 30556 consensus_queue.cc:1035] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:57.726524 31096 ts_tablet_manager.cc:1398] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240502 08:17:57.727735 31156 tablet_bootstrap.cc:492] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:57.728003 31156 ts_tablet_manager.cc:1367] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.174s user 0.000s sys 0.004s
I20240502 08:17:57.728919 31156 raft_consensus.cc:348] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:57.729238 31156 raft_consensus.cc:374] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.729362 31156 raft_consensus.cc:729] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.729681 31156 consensus_queue.cc:260] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:57.729820 31048 raft_consensus.cc:2932] T db55b42c724246da8d9cb23369826279 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:17:57.730418 27390 raft_consensus.cc:2932] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:17:57.731439 31156 ts_tablet_manager.cc:1398] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:17:57.736368 31117 ts_tablet_manager.cc:1367] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.100s user 0.007s sys 0.001s
I20240502 08:17:57.737289 31137 ts_tablet_manager.cc:1888] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:57.737533 31137 log.cc:1196] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/a67cb9c98522464399a44186109333cf
I20240502 08:17:57.738070 31137 ts_tablet_manager.cc:1909] T a67cb9c98522464399a44186109333cf P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:57.738329 27107 catalog_manager.cc:5484] T db55b42c724246da8d9cb23369826279 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:17:57.739306 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet db55b42c724246da8d9cb23369826279 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:57.744130 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet db55b42c724246da8d9cb23369826279 with delete_type TABLET_DATA_TOMBSTONED (TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:57.744700 30158 raft_consensus.cc:2932] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:17:57.745450 30524 tablet_replica.cc:330] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:57.745644 30524 raft_consensus.cc:2229] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:57.745857 30524 raft_consensus.cc:2258] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:57.746594 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet a67cb9c98522464399a44186109333cf (table rebalance_test_table_18 [id=9df8e288a677400882b593a841c5c051]) successfully deleted
I20240502 08:17:57.746721 31137 tablet_replica.cc:330] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:57.746899 31137 raft_consensus.cc:2229] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:57.747164 31137 raft_consensus.cc:2258] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:57.747253 30524 ts_tablet_manager.cc:1875] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:57.747790 31137 ts_tablet_manager.cc:1875] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:57.748368 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet c2776f857a5f47e9b8022639dc54757d from peer 49c5c8f70e494edd9fcda40cc0af6cc4 ({username='slave'} at 127.20.207.7:58763)
I20240502 08:17:57.748539 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet c2776f857a5f47e9b8022639dc54757d from peer 49c5c8f70e494edd9fcda40cc0af6cc4 at {username='slave'} at 127.20.207.7:58763: session id = 49c5c8f70e494edd9fcda40cc0af6cc4-c2776f857a5f47e9b8022639dc54757d
I20240502 08:17:57.751334 27541 tablet_copy_source_session.cc:205] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:57.753595 31096 tablet_bootstrap.cc:492] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:57.753782 31167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2776f857a5f47e9b8022639dc54757d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.755167 30524 ts_tablet_manager.cc:1888] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:17:57.755287 30524 log.cc:1196] T db55b42c724246da8d9cb23369826279 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/db55b42c724246da8d9cb23369826279
I20240502 08:17:57.756031 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet db55b42c724246da8d9cb23369826279 (table rebalance_test_table_13 [id=28a67fad5bff471981e8cdf2b2a2b34a]) successfully deleted
I20240502 08:17:57.756088 27107 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:17:57.756453 31096 tablet_bootstrap.cc:654] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.756685 30968 raft_consensus.cc:482] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:57.756829 30968 raft_consensus.cc:504] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:57.757566 30968 leader_election.cc:288] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:57.759541 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26545a868bd6415c9c9db4869b2841c7" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:17:57.759740 30160 raft_consensus.cc:2454] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:17:57.760133 29964 leader_election.cc:302] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:57.760397 30968 raft_consensus.cc:2781] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:57.760501 30968 raft_consensus.cc:482] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:57.760579 30968 raft_consensus.cc:3037] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.761840 31117 raft_consensus.cc:348] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:17:57.762084 31117 raft_consensus.cc:729] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: LEARNER
I20240502 08:17:57.762344 31117 consensus_queue.cc:260] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:17:57.762969 31117 ts_tablet_manager.cc:1398] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240502 08:17:57.763404 30968 raft_consensus.cc:504] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:57.764305 30968 leader_election.cc:288] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:57.764380 30158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26545a868bd6415c9c9db4869b2841c7" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:17:57.764542 30158 raft_consensus.cc:3037] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.767060 30158 raft_consensus.cc:2454] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:17:57.767460 29964 leader_election.cc:302] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:57.767838 30968 raft_consensus.cc:2781] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:57.767987 30968 raft_consensus.cc:686] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:17:57.768301 30968 consensus_queue.cc:237] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:57.765935 31137 ts_tablet_manager.cc:1888] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:17:57.772738 31137 log.cc:1196] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/549102ad342342f49049f34ebac5bc7d
I20240502 08:17:57.773607 31137 ts_tablet_manager.cc:1909] T 549102ad342342f49049f34ebac5bc7d P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:57.774235 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-81e73229c0d145e7bb81761bf4b820dc received from {username='slave'} at 127.20.207.2:44351
I20240502 08:17:57.774394 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-81e73229c0d145e7bb81761bf4b820dc on tablet 81e73229c0d145e7bb81761bf4b820dc with peer 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:17:57.776618 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 549102ad342342f49049f34ebac5bc7d (table rebalance_test_table_18 [id=9df8e288a677400882b593a841c5c051]) successfully deleted
I20240502 08:17:57.778645 31146 tablet.cc:1722] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:57.779052 27113 catalog_manager.cc:5484] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:57.783020 30611 raft_consensus.cc:482] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:57.783170 30611 raft_consensus.cc:504] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:57.784026 30611 leader_election.cc:288] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:17:57.785744 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84953bd8daba4885be1f5f9ad4149aba" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:17:57.785972 30288 raft_consensus.cc:2454] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:57.786615 27584 leader_election.cc:302] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:57.786898 30611 raft_consensus.cc:2781] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:57.786998 30611 raft_consensus.cc:482] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:57.787060 30611 raft_consensus.cc:3037] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.788403 31147 tablet.cc:1722] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:57.789378 31154 tablet.cc:1722] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:57.789610 30611 raft_consensus.cc:504] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:57.792325 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84953bd8daba4885be1f5f9ad4149aba" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:17:57.792555 30288 raft_consensus.cc:3037] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.795266 30288 raft_consensus.cc:2454] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:57.795686 27584 leader_election.cc:302] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:17:57.798839 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=885b6edfa1754349a3060bb856f6f829] alter complete (version 3)
I20240502 08:17:57.802958 30611 leader_election.cc:288] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:17:57.803882 30611 raft_consensus.cc:2781] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:57.804433 30611 raft_consensus.cc:686] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:17:57.805063 30611 consensus_queue.cc:237] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:57.807376 31167 tablet_copy_client.cc:783] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:57.807749 31167 tablet_copy_client.cc:647] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:57.810370 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26545a868bd6415c9c9db4869b2841c7" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:17:57.810739 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26545a868bd6415c9c9db4869b2841c7" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
W20240502 08:17:57.811440 29961 leader_election.cc:341] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:17:57.812455 29961 leader_election.cc:341] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:57.816026 27390 raft_consensus.cc:1203] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:17:57.818150 31167 tablet_copy_client.cc:523] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:57.822788 31167 tablet_bootstrap.cc:492] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:57.828369 31096 tablet_bootstrap.cc:492] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:57.833089 31096 ts_tablet_manager.cc:1367] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.104s user 0.002s sys 0.004s
I20240502 08:17:57.837177 31096 raft_consensus.cc:348] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:57.837504 31096 raft_consensus.cc:374] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.838069 31096 raft_consensus.cc:729] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.839350 31096 consensus_queue.cc:260] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:57.839895 30158 raft_consensus.cc:1203] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:57.844786 27390 raft_consensus.cc:1203] T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:17:57.847766 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84953bd8daba4885be1f5f9ad4149aba" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:17:57.848007 30028 raft_consensus.cc:2454] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:17:57.848659 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84953bd8daba4885be1f5f9ad4149aba" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:17:57.848807 30028 raft_consensus.cc:3037] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.851601 30028 raft_consensus.cc:2454] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:17:57.854496 27112 catalog_manager.cc:5484] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:57.858904 31096 ts_tablet_manager.cc:1398] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240502 08:17:57.901517 27113 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:17:57.901747 27113 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:17:57.903522 30981 raft_consensus.cc:482] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:57.903654 30981 raft_consensus.cc:504] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:57.904173 30981 leader_election.cc:288] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:17:57.905174 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5a0175799e478c9f53ecc82ba090ae" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:17:57.905428 30422 raft_consensus.cc:2454] T 0a5a0175799e478c9f53ecc82ba090ae P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:17:57.905802 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5a0175799e478c9f53ecc82ba090ae" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:17:57.906131 30093 leader_election.cc:302] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:57.906399 30290 raft_consensus.cc:2454] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:17:57.906783 27113 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet e8a610177b03466eb454c731ac756f1e
I20240502 08:17:57.906940 30981 raft_consensus.cc:2781] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:57.907097 30981 raft_consensus.cc:482] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:57.907171 30981 raft_consensus.cc:3037] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.907857 27370 tablet_service.cc:1514] Processing DeleteTablet for tablet e8a610177b03466eb454c731ac756f1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:57 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:17:57.908056 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet e8a610177b03466eb454c731ac756f1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:57 UTC) from {username='slave'} at 127.0.0.1:52730
I20240502 08:17:57.908234 31137 tablet_replica.cc:330] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:17:57.908454 31137 raft_consensus.cc:2229] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:57.908820 31137 raft_consensus.cc:2258] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:57.909725 31137 ts_tablet_manager.cc:1875] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:57.909811 30981 raft_consensus.cc:504] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:57.916982 31137 ts_tablet_manager.cc:1888] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:57.917133 31137 log.cc:1196] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/e8a610177b03466eb454c731ac756f1e
I20240502 08:17:57.917620 31137 ts_tablet_manager.cc:1909] T e8a610177b03466eb454c731ac756f1e P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:17:57.918282 27390 consensus_queue.cc:237] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:17:57.919703 30401 tablet_service.cc:1514] Processing DeleteTablet for tablet e8a610177b03466eb454c731ac756f1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:57 UTC) from {username='slave'} at 127.0.0.1:56694
I20240502 08:17:57.920181 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5a0175799e478c9f53ecc82ba090ae" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:17:57.920209 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet e8a610177b03466eb454c731ac756f1e (table rebalancer_dynamic_table_new_name [id=885b6edfa1754349a3060bb856f6f829]) successfully deleted
I20240502 08:17:57.920379 30422 raft_consensus.cc:3037] T 0a5a0175799e478c9f53ecc82ba090ae P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.921789 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5a0175799e478c9f53ecc82ba090ae" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:17:57.922102 30290 raft_consensus.cc:3037] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.922972 30422 raft_consensus.cc:2454] T 0a5a0175799e478c9f53ecc82ba090ae P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:17:57.924893 30093 leader_election.cc:302] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:57.925170 30981 leader_election.cc:288] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:17:57.925209 31113 raft_consensus.cc:2781] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:57.925423 31113 raft_consensus.cc:686] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:17:57.925722 31113 consensus_queue.cc:237] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:57.931794 30290 raft_consensus.cc:2454] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:17:57.933271 31167 tablet_bootstrap.cc:492] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:57.933868 31167 tablet_bootstrap.cc:492] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap complete.
I20240502 08:17:57.934267 31167 ts_tablet_manager.cc:1367] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.112s user 0.057s sys 0.000s
I20240502 08:17:57.935390 31167 raft_consensus.cc:348] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:17:57.935748 31167 raft_consensus.cc:729] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: LEARNER
I20240502 08:17:57.936290 31167 consensus_queue.cc:260] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:17:57.938097 31167 ts_tablet_manager.cc:1398] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:57.938812 31202 tablet_replica.cc:330] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:17:57.939111 31202 raft_consensus.cc:2229] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:57.939392 31202 raft_consensus.cc:2258] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:57.940454 31200 tablet_replica.cc:330] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:17:57.940663 31200 raft_consensus.cc:2229] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:57.940917 31200 raft_consensus.cc:2258] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:57.941184 31202 ts_tablet_manager.cc:1875] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:57.941762 31200 ts_tablet_manager.cc:1875] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:57.943478 30422 raft_consensus.cc:1261] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:57.944487 30028 raft_consensus.cc:1261] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:17:57.946543 27326 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Couldn't send request to peer bf8ac71b15f740a0a928911360773442. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6a26e01d42b429694641f0ceca11265. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:57.947466 31050 consensus_queue.cc:1035] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:17:57.947656 31202 ts_tablet_manager.cc:1888] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:57.947773 31202 log.cc:1196] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/e8a610177b03466eb454c731ac756f1e
I20240502 08:17:57.948032 31200 ts_tablet_manager.cc:1888] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:57.948137 31200 log.cc:1196] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/e8a610177b03466eb454c731ac756f1e
I20240502 08:17:57.948223 31202 ts_tablet_manager.cc:1909] T e8a610177b03466eb454c731ac756f1e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting consensus metadata
I20240502 08:17:57.948215 31050 consensus_queue.cc:1035] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:17:57.948561 31200 ts_tablet_manager.cc:1909] T e8a610177b03466eb454c731ac756f1e P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting consensus metadata
I20240502 08:17:57.949257 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet e8a610177b03466eb454c731ac756f1e (table rebalancer_dynamic_table_new_name [id=885b6edfa1754349a3060bb856f6f829]) successfully deleted
I20240502 08:17:57.949469 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-c2776f857a5f47e9b8022639dc54757d received from {username='slave'} at 127.20.207.7:58763
I20240502 08:17:57.949584 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-c2776f857a5f47e9b8022639dc54757d on tablet c2776f857a5f47e9b8022639dc54757d with peer 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:17:57.950378 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet e8a610177b03466eb454c731ac756f1e (table rebalancer_dynamic_table_new_name [id=885b6edfa1754349a3060bb856f6f829]) successfully deleted
I20240502 08:17:57.960973 27112 catalog_manager.cc:5484] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:57.963169 30826 raft_consensus.cc:2932] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:17:57.965293 30422 raft_consensus.cc:2932] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:17:57.965342 30028 raft_consensus.cc:2932] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:17:57.970464 27112 catalog_manager.cc:5484] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:57.983898 30160 consensus_queue.cc:237] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:17:57.986045 27652 raft_consensus.cc:1261] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:57.986760 30290 raft_consensus.cc:1261] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:17:57.986793 30095 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 -> Peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): Couldn't send request to peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec6c977cac054b758a43d846e821707a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:57.987315 30981 consensus_queue.cc:1035] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:17:57.987946 31110 consensus_queue.cc:1035] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:17:57.989606 31110 raft_consensus.cc:2932] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:17:57.990694 30290 raft_consensus.cc:2932] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:17:57.990819 27650 raft_consensus.cc:2932] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:17:57.995715 27107 catalog_manager.cc:5484] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:58.008801 27258 consensus_queue.cc:237] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:17:58.009280 27110 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:17:58.018064 30401 tablet_service.cc:1467] Processing CreateTablet for tablet 8c75aa1b58874966994f6cfa7258505a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=07e21a6cd16e4614b0a3130f7a9219e4]), partition=
I20240502 08:17:58.018425 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c75aa1b58874966994f6cfa7258505a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.018421 30139 tablet_service.cc:1467] Processing CreateTablet for tablet 8c75aa1b58874966994f6cfa7258505a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=07e21a6cd16e4614b0a3130f7a9219e4]), partition=
I20240502 08:17:58.018579 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 8c75aa1b58874966994f6cfa7258505a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=07e21a6cd16e4614b0a3130f7a9219e4]), partition=
I20240502 08:17:58.018888 30139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c75aa1b58874966994f6cfa7258505a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.018949 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c75aa1b58874966994f6cfa7258505a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:58.024911 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2dfa509d47e54b0f9ccacc9954981fb6. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:58.025667 31096 tablet_bootstrap.cc:492] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:58.026012 27520 raft_consensus.cc:1261] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:58.026666 27390 raft_consensus.cc:1261] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:58.027267 31096 tablet_bootstrap.cc:654] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:58.027281 30871 consensus_queue.cc:1035] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:58.027635 31009 consensus_queue.cc:1035] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:58.031360 31161 tablet_bootstrap.cc:492] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:58.032804 31161 tablet_bootstrap.cc:654] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:58.033250 31009 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:17:58.033799 27390 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:17:58.043576 27112 catalog_manager.cc:5484] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:58.054471 31219 tablet_bootstrap.cc:492] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:17:58.056529 31219 tablet_bootstrap.cc:654] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:58.058326 31096 tablet_bootstrap.cc:492] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:58.058807 31096 ts_tablet_manager.cc:1367] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.005s
I20240502 08:17:58.059684 31161 tablet_bootstrap.cc:492] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:17:58.059888 31161 ts_tablet_manager.cc:1367] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.004s
I20240502 08:17:58.060662 31161 raft_consensus.cc:348] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:58.060925 31161 raft_consensus.cc:374] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:58.061061 31161 raft_consensus.cc:729] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:17:58.061363 31161 consensus_queue.cc:260] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:58.065023 31161 ts_tablet_manager.cc:1398] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:17:58.068310 31096 raft_consensus.cc:348] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:58.068575 31096 raft_consensus.cc:374] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:58.068470 27650 raft_consensus.cc:1203] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:17:58.068688 31096 raft_consensus.cc:729] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:58.070437 31096 consensus_queue.cc:260] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:58.070941 31096 ts_tablet_manager.cc:1398] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:58.073889 31219 tablet_bootstrap.cc:492] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:17:58.074127 31219 ts_tablet_manager.cc:1367] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.000s
I20240502 08:17:58.074738 27521 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:17:58.075142 31219 raft_consensus.cc:348] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:58.075361 31219 raft_consensus.cc:374] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:58.075448 31219 raft_consensus.cc:729] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:17:58.075695 31219 consensus_queue.cc:260] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:58.080636 31219 ts_tablet_manager.cc:1398] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:17:58.189623 31144 raft_consensus.cc:482] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:58.189803 31144 raft_consensus.cc:504] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:58.190645 31144 leader_election.cc:288] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:17:58.191073 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c75aa1b58874966994f6cfa7258505a" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:17:58.191124 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c75aa1b58874966994f6cfa7258505a" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:17:58.191272 30160 raft_consensus.cc:2454] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:17:58.191340 30290 raft_consensus.cc:2454] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:17:58.191664 30358 leader_election.cc:302] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:58.192190 31144 raft_consensus.cc:2781] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:58.192304 31144 raft_consensus.cc:482] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:58.192376 31144 raft_consensus.cc:3037] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:58.197780 31144 raft_consensus.cc:504] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:58.198529 31144 leader_election.cc:288] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:17:58.198861 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c75aa1b58874966994f6cfa7258505a" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:17:58.199041 30160 raft_consensus.cc:3037] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:58.199188 30653 raft_consensus.cc:482] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:58.199266 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c75aa1b58874966994f6cfa7258505a" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:17:58.199359 30653 raft_consensus.cc:504] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:58.199470 30290 raft_consensus.cc:3037] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:58.200187 30653 leader_election.cc:288] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:58.202184 30290 raft_consensus.cc:2454] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:17:58.202205 30160 raft_consensus.cc:2454] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:17:58.202530 30355 leader_election.cc:302] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:17:58.204695 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26545a868bd6415c9c9db4869b2841c7" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:17:58.205770 31144 raft_consensus.cc:2781] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:58.206095 31144 raft_consensus.cc:686] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:17:58.206547 31144 consensus_queue.cc:237] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:58.214675 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26545a868bd6415c9c9db4869b2841c7" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:17:58.214957 30160 raft_consensus.cc:2379] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in current term 1: Already voted for candidate 1f0bf4bc3c5146539146b825b048bff3 in this term.
I20240502 08:17:58.215402 30226 leader_election.cc:302] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4; no voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6
I20240502 08:17:58.216012 30653 raft_consensus.cc:3037] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:58.218771 30653 raft_consensus.cc:2726] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:58.235129 30968 consensus_queue.cc:1035] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:58.247225 27108 catalog_manager.cc:5484] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:58.261389 30653 raft_consensus.cc:1050] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4: attempting to promote NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 to VOTER
I20240502 08:17:58.262457 30653 consensus_queue.cc:237] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:17:58.264904 31141 consensus_queue.cc:1035] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:58.268652 27390 raft_consensus.cc:1261] T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:58.268833 30028 raft_consensus.cc:1261] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:58.269701 30422 raft_consensus.cc:1261] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:58.270079 31024 consensus_queue.cc:1035] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:17:58.270776 31024 consensus_queue.cc:1035] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:17:58.271322 31024 consensus_queue.cc:1035] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:17:58.282850 30653 raft_consensus.cc:2932] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:17:58.284237 30871 raft_consensus.cc:1050] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER 84cda20738d2452da166e31fc6da57a6 to VOTER
I20240502 08:17:58.285041 30871 consensus_queue.cc:237] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:17:58.288309 30028 raft_consensus.cc:2932] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:17:58.289049 27390 raft_consensus.cc:2932] T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:17:58.294458 27112 catalog_manager.cc:5484] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:17:58.295583 30422 raft_consensus.cc:2932] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:17:58.306349 30290 consensus_queue.cc:237] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:17:58.307663 31009 raft_consensus.cc:1050] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 to VOTER
I20240502 08:17:58.308588 27390 raft_consensus.cc:1261] T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:58.308689 31009 consensus_queue.cc:237] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:17:58.309082 31266 consensus_queue.cc:1035] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:58.311223 30422 raft_consensus.cc:1261] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:58.311668 31266 consensus_queue.cc:1035] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:58.317785 30653 raft_consensus.cc:2932] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:17:58.318395 27388 raft_consensus.cc:2932] T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:17:58.319061 30422 raft_consensus.cc:2932] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:17:58.321583 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dd644c6742854b79a212d858fa9df147 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:58.323161 27110 catalog_manager.cc:5484] T dd644c6742854b79a212d858fa9df147 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:58.324128 27521 raft_consensus.cc:1261] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:58.324470 27650 raft_consensus.cc:1261] T bd640e48563f458cb2d56c94ed1e0347 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:58.324657 30871 consensus_queue.cc:1035] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:58.325181 30871 consensus_queue.cc:1035] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:58.325361 30158 raft_consensus.cc:1261] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:58.326809 30871 consensus_queue.cc:1035] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:58.330930 27238 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 08:17:58.333086 27388 raft_consensus.cc:1261] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:58.333232 30158 raft_consensus.cc:1261] T 81e73229c0d145e7bb81761bf4b820dc P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:58.333945 30871 consensus_queue.cc:1035] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:17:58.334348 31012 consensus_queue.cc:1035] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:17:58.342252 30008 tablet_service.cc:1514] Processing DeleteTablet for tablet dd644c6742854b79a212d858fa9df147 with delete_type TABLET_DATA_TOMBSTONED (TS 1f0bf4bc3c5146539146b825b048bff3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58374
I20240502 08:17:58.346850 30290 raft_consensus.cc:1203] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:58.368564 30611 consensus_queue.cc:1035] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:58.384371 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
W20240502 08:17:58.386675 31267 log.cc:924] Time spent T dd644c6742854b79a212d858fa9df147 P 83e171a5e16c4008923f1d43fc5ce7b1: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20240502 08:17:58.404304 30290 raft_consensus.cc:1261] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:58.405062 31009 consensus_queue.cc:1035] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:17:58.412171 31012 raft_consensus.cc:2932] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:17:58.416159 31110 consensus_queue.cc:1035] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:58.421111 27390 raft_consensus.cc:2932] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:17:58.429030 27112 catalog_manager.cc:5484] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:17:58.439482 30288 raft_consensus.cc:2932] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:17:58.448259 31167 ts_tablet_manager.cc:902] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:58.451058 31279 tablet_replica.cc:330] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:17:58.451431 31279 raft_consensus.cc:2229] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:58.452288 31279 raft_consensus.cc:2258] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:58.453935 31279 ts_tablet_manager.cc:1875] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:58.460090 31110 consensus_queue.cc:1035] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:58.466675 30611 consensus_queue.cc:1035] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:58.472566 27108 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:58.484831 30006 tablet_service.cc:1467] Processing CreateTablet for tablet d00f25f29f5d4db2a2261dd37c0793fa (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=e2a08da4858541088d7e3e5a7ae41e45]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:58.485271 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00f25f29f5d4db2a2261dd37c0793fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.487226 30139 tablet_service.cc:1467] Processing CreateTablet for tablet d00f25f29f5d4db2a2261dd37c0793fa (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=e2a08da4858541088d7e3e5a7ae41e45]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:58.487735 30139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00f25f29f5d4db2a2261dd37c0793fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.488193 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 0dea1ac1af1b45cf9abfd5e5a1d910a1 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=e2a08da4858541088d7e3e5a7ae41e45]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:58.488646 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dea1ac1af1b45cf9abfd5e5a1d910a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.494233 31096 tablet_bootstrap.cc:492] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:58.495648 30138 tablet_service.cc:1467] Processing CreateTablet for tablet 57505255575249e698bf35929619423d (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=e2a08da4858541088d7e3e5a7ae41e45]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:58.495988 31096 tablet_bootstrap.cc:654] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:58.495998 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57505255575249e698bf35929619423d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.497851 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 57505255575249e698bf35929619423d (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=e2a08da4858541088d7e3e5a7ae41e45]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:58.498313 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57505255575249e698bf35929619423d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.499217 30007 tablet_service.cc:1467] Processing CreateTablet for tablet 57505255575249e698bf35929619423d (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=e2a08da4858541088d7e3e5a7ae41e45]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:58.499619 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57505255575249e698bf35929619423d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.503772 31161 tablet_bootstrap.cc:492] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:58.504549 30008 tablet_service.cc:1467] Processing CreateTablet for tablet 0dea1ac1af1b45cf9abfd5e5a1d910a1 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=e2a08da4858541088d7e3e5a7ae41e45]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:58.505061 30008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dea1ac1af1b45cf9abfd5e5a1d910a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.505477 31161 tablet_bootstrap.cc:654] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:58.506850 30005 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:17:58.512167 27194 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.3:44469, user_credentials={real_user=slave}} blocked reactor thread for 92826.9us
I20240502 08:17:58.514598 30871 raft_consensus.cc:2932] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:17:58.530942 31279 ts_tablet_manager.cc:1888] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:17:58.531114 31279 log.cc:1196] T dd644c6742854b79a212d858fa9df147 P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/dd644c6742854b79a212d858fa9df147
I20240502 08:17:58.531795 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet dd644c6742854b79a212d858fa9df147 (table rebalancer_extra_table_2 [id=e14b3e831e834278ac631fa1cb84dced]) successfully deleted
I20240502 08:17:58.532666 30158 raft_consensus.cc:2932] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:17:58.536845 30159 raft_consensus.cc:1261] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:58.538336 31259 consensus_queue.cc:1035] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:58.539224 31021 raft_consensus.cc:1050] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4: attempting to promote NON_VOTER bf8ac71b15f740a0a928911360773442 to VOTER
I20240502 08:17:58.541126 30288 raft_consensus.cc:1261] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:58.542135 31259 consensus_queue.cc:1035] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:58.543473 31161 tablet_bootstrap.cc:492] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:17:58.543696 31161 ts_tablet_manager.cc:1367] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.003s
I20240502 08:17:58.544430 31161 raft_consensus.cc:348] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:58.544201 27108 catalog_manager.cc:5484] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:17:58.544785 31161 raft_consensus.cc:374] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:58.540028 31021 consensus_queue.cc:237] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:17:58.544982 31161 raft_consensus.cc:729] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:17:58.545282 31161 consensus_queue.cc:260] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:58.545868 31161 ts_tablet_manager.cc:1398] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:58.546160 31161 tablet_bootstrap.cc:492] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:58.547796 31161 tablet_bootstrap.cc:654] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:58.551327 27652 raft_consensus.cc:2932] T bd640e48563f458cb2d56c94ed1e0347 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:17:58.564127 27520 raft_consensus.cc:2932] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:17:58.566603 27631 tablet_service.cc:1467] Processing CreateTablet for tablet d00f25f29f5d4db2a2261dd37c0793fa (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=e2a08da4858541088d7e3e5a7ae41e45]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:58.567073 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00f25f29f5d4db2a2261dd37c0793fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.571857 30160 raft_consensus.cc:2932] T 81e73229c0d145e7bb81761bf4b820dc P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:17:58.574036 30401 tablet_service.cc:1467] Processing CreateTablet for tablet 0dea1ac1af1b45cf9abfd5e5a1d910a1 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=e2a08da4858541088d7e3e5a7ae41e45]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:58.574472 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dea1ac1af1b45cf9abfd5e5a1d910a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.576316 27256 consensus_queue.cc:237] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:17:58.576552 31305 ts_tablet_manager.cc:902] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Initiating tablet copy from peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:58.578806 27388 raft_consensus.cc:1261] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:58.579933 31012 consensus_queue.cc:1035] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:58.584772 31315 ts_tablet_manager.cc:902] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442: Initiating tablet copy from peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:58.587486 31009 raft_consensus.cc:2932] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:17:58.587677 27388 raft_consensus.cc:2932] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:17:58.590257 27370 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:58.594063 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 81e73229c0d145e7bb81761bf4b820dc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:58.595804 27116 catalog_manager.cc:5484] T 81e73229c0d145e7bb81761bf4b820dc P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:17:58.600476 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet 81e73229c0d145e7bb81761bf4b820dc with delete_type TABLET_DATA_TOMBSTONED (TS 49c5c8f70e494edd9fcda40cc0af6cc4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52730
I20240502 08:17:58.600845 27258 raft_consensus.cc:1261] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:58.603988 27110 catalog_manager.cc:5576] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899) does not have the latest schema for tablet 8c75aa1b58874966994f6cfa7258505a (table rebalancer_dynamic_table [id=07e21a6cd16e4614b0a3130f7a9219e4]). Expected version 1 got 0
I20240502 08:17:58.605046 31161 tablet_bootstrap.cc:492] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:17:58.605468 31308 tablet_bootstrap.cc:492] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:58.607621 31161 ts_tablet_manager.cc:1367] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.062s user 0.006s sys 0.000s
I20240502 08:17:58.608904 31308 tablet_bootstrap.cc:654] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:58.613216 27650 raft_consensus.cc:1261] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:58.613108 27258 consensus_queue.cc:237] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:17:58.617952 27650 raft_consensus.cc:1261] T bd640e48563f458cb2d56c94ed1e0347 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:58.618682 31294 consensus_queue.cc:1035] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:58.618762 30157 raft_consensus.cc:1261] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:58.619531 31161 raft_consensus.cc:348] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:17:58.620096 31161 raft_consensus.cc:374] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:58.620265 31161 raft_consensus.cc:729] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:17:58.620338 30421 raft_consensus.cc:1261] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:58.620589 31161 consensus_queue.cc:260] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:17:58.621320 31161 ts_tablet_manager.cc:1398] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 08:17:58.622213 31009 consensus_queue.cc:1035] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:58.624223 31009 raft_consensus.cc:2932] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:17:58.627498 30139 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:58.628650 30159 raft_consensus.cc:1261] T 81e73229c0d145e7bb81761bf4b820dc P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:58.628568 27650 raft_consensus.cc:2932] T bd640e48563f458cb2d56c94ed1e0347 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:17:58.630048 30871 consensus_queue.cc:1035] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:58.631014 30157 raft_consensus.cc:2932] T bd640e48563f458cb2d56c94ed1e0347 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:17:58.631672 30159 raft_consensus.cc:2932] T 81e73229c0d145e7bb81761bf4b820dc P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:17:58.636467 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bd640e48563f458cb2d56c94ed1e0347 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:58.641185 27116 catalog_manager.cc:5484] T bd640e48563f458cb2d56c94ed1e0347 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:58.641630 31021 consensus_queue.cc:1035] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20240502 08:17:58.642388 31325 tablet.cc:1722] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:58.644532 31024 consensus_queue.cc:1035] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20240502 08:17:58.644897 31325 tablet.cc:1620] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:17:58.645655 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet bd640e48563f458cb2d56c94ed1e0347 with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52776
I20240502 08:17:58.647058 27111 catalog_manager.cc:5576] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899) does not have the latest schema for tablet 8c75aa1b58874966994f6cfa7258505a (table rebalancer_dynamic_table [id=07e21a6cd16e4614b0a3130f7a9219e4]). Expected version 1 got 0
I20240502 08:17:58.650444 30653 consensus_queue.cc:1035] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:17:58.654700 27109 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet 8c75aa1b58874966994f6cfa7258505a (table rebalancer_dynamic_table [id=07e21a6cd16e4614b0a3130f7a9219e4]). Expected version 1 got 0
I20240502 08:17:58.656692 31096 tablet_bootstrap.cc:492] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:58.657063 31096 ts_tablet_manager.cc:1367] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.163s user 0.003s sys 0.011s
I20240502 08:17:58.658000 31096 raft_consensus.cc:348] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:58.658363 31096 raft_consensus.cc:374] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:58.658495 31096 raft_consensus.cc:729] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:58.658812 31096 consensus_queue.cc:260] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:58.659340 31096 ts_tablet_manager.cc:1398] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:58.659626 31096 tablet_bootstrap.cc:492] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:58.661167 31096 tablet_bootstrap.cc:654] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:58.662446 31322 tablet_replica.cc:330] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:17:58.662664 31322 raft_consensus.cc:2229] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:58.663044 31322 raft_consensus.cc:2258] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:58.663820 31308 tablet_bootstrap.cc:492] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:58.665529 31320 tablet.cc:1722] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:58.665859 30268 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:58.666355 31328 tablet_replica.cc:330] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:17:58.666689 31328 raft_consensus.cc:2229] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:58.667129 31328 raft_consensus.cc:2258] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:58.667829 31308 ts_tablet_manager.cc:1367] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.000s
I20240502 08:17:58.668332 31328 ts_tablet_manager.cc:1875] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:58.669018 31308 raft_consensus.cc:348] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:17:58.672605 31308 raft_consensus.cc:374] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:58.676913 31308 raft_consensus.cc:729] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:58.677304 31308 consensus_queue.cc:260] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:17:58.677819 27111 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet 8c75aa1b58874966994f6cfa7258505a (table rebalancer_dynamic_table [id=07e21a6cd16e4614b0a3130f7a9219e4]). Expected version 1 got 0
I20240502 08:17:58.679898 31308 ts_tablet_manager.cc:1398] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 08:17:58.680567 30653 raft_consensus.cc:2932] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:17:58.681896 27116 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet 8c75aa1b58874966994f6cfa7258505a (table rebalancer_dynamic_table [id=07e21a6cd16e4614b0a3130f7a9219e4]). Expected version 1 got 0
I20240502 08:17:58.682574 31308 tablet_bootstrap.cc:492] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:58.682233 27652 raft_consensus.cc:2932] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:17:58.684269 31320 tablet.cc:1620] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:17:58.686091 31322 ts_tablet_manager.cc:1875] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:58.686156 27258 raft_consensus.cc:2932] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:17:58.688076 30401 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:58.689076 27629 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:58.669327 31321 tablet_bootstrap.cc:492] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:17:58.693295 31321 tablet_bootstrap.cc:654] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:58.695583 31308 tablet_bootstrap.cc:654] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:58.700924 27111 catalog_manager.cc:5484] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:17:58.704016 27108 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=07e21a6cd16e4614b0a3130f7a9219e4] alter complete (version 1)
I20240502 08:17:58.705665 31308 tablet_bootstrap.cc:492] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:58.708463 31308 ts_tablet_manager.cc:1367] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240502 08:17:58.710048 31308 raft_consensus.cc:348] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:58.711061 31308 raft_consensus.cc:374] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:58.711305 31308 raft_consensus.cc:729] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:58.711789 31308 consensus_queue.cc:260] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:58.714326 31308 ts_tablet_manager.cc:1398] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:17:58.716660 31308 tablet_bootstrap.cc:492] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:58.719658 31308 tablet_bootstrap.cc:654] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:58.722896 31316 tablet_bootstrap.cc:492] T d00f25f29f5d4db2a2261dd37c0793fa P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:58.724375 31316 tablet_bootstrap.cc:654] T d00f25f29f5d4db2a2261dd37c0793fa P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:58.726748 31167 tablet_copy_client.cc:308] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:17:58.727490 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet 2dfa509d47e54b0f9ccacc9954981fb6 from peer 49c5c8f70e494edd9fcda40cc0af6cc4 ({username='slave'} at 127.20.207.7:45231)
I20240502 08:17:58.727658 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet 2dfa509d47e54b0f9ccacc9954981fb6 from peer 49c5c8f70e494edd9fcda40cc0af6cc4 at {username='slave'} at 127.20.207.7:45231: session id = 49c5c8f70e494edd9fcda40cc0af6cc4-2dfa509d47e54b0f9ccacc9954981fb6
I20240502 08:17:58.730659 27278 tablet_copy_source_session.cc:205] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:58.732760 31167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfa509d47e54b0f9ccacc9954981fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.732750 27500 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:17:58.738098 31305 tablet_copy_client.cc:308] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.6:38899
I20240502 08:17:58.739240 30180 tablet_copy_service.cc:140] P 84cda20738d2452da166e31fc6da57a6: Received BeginTabletCopySession request for tablet ec6c977cac054b758a43d846e821707a from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 ({username='slave'} at 127.20.207.1:35051)
I20240502 08:17:58.739470 30180 tablet_copy_service.cc:161] P 84cda20738d2452da166e31fc6da57a6: Beginning new tablet copy session on tablet ec6c977cac054b758a43d846e821707a from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 at {username='slave'} at 127.20.207.1:35051: session id = 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-ec6c977cac054b758a43d846e821707a
I20240502 08:17:58.741343 31308 tablet_bootstrap.cc:492] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:58.741474 31308 ts_tablet_manager.cc:1367] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.002s
I20240502 08:17:58.742303 31308 raft_consensus.cc:348] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:58.742573 31308 raft_consensus.cc:374] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:58.742689 31308 raft_consensus.cc:729] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:58.742967 31308 consensus_queue.cc:260] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:58.743330 30180 tablet_copy_source_session.cc:205] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:58.746472 31327 tablet.cc:1722] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:58.747162 31327 tablet.cc:1620] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:17:58.747215 31305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6c977cac054b758a43d846e821707a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.748263 31308 ts_tablet_manager.cc:1398] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 08:17:58.750303 31096 tablet_bootstrap.cc:492] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:58.750479 31096 ts_tablet_manager.cc:1367] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.091s user 0.005s sys 0.000s
I20240502 08:17:58.751339 31167 tablet_copy_client.cc:783] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:58.751298 31096 raft_consensus.cc:348] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:17:58.751607 31167 tablet_copy_client.cc:647] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:58.751667 31096 raft_consensus.cc:374] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:58.751811 31096 raft_consensus.cc:729] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:58.752116 31096 consensus_queue.cc:260] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:17:58.752712 31096 ts_tablet_manager.cc:1398] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:58.754724 31167 tablet_copy_client.cc:523] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:58.757962 31167 tablet_bootstrap.cc:492] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:58.759780 31328 ts_tablet_manager.cc:1888] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:17:58.761087 31328 log.cc:1196] T bd640e48563f458cb2d56c94ed1e0347 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/bd640e48563f458cb2d56c94ed1e0347
I20240502 08:17:58.766857 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet bd640e48563f458cb2d56c94ed1e0347 (table rebalance_test_table_5 [id=5c5af31071894d4f803992d5173e0429]) successfully deleted
I20240502 08:17:58.767606 31315 tablet_copy_client.cc:308] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.2:46291
I20240502 08:17:58.768406 31322 ts_tablet_manager.cc:1888] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:17:58.768599 31322 log.cc:1196] T 81e73229c0d145e7bb81761bf4b820dc P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/81e73229c0d145e7bb81761bf4b820dc
I20240502 08:17:58.768543 27410 tablet_copy_service.cc:140] P 83e171a5e16c4008923f1d43fc5ce7b1: Received BeginTabletCopySession request for tablet b6a26e01d42b429694641f0ceca11265 from peer bf8ac71b15f740a0a928911360773442 ({username='slave'} at 127.20.207.4:39607)
I20240502 08:17:58.768766 27410 tablet_copy_service.cc:161] P 83e171a5e16c4008923f1d43fc5ce7b1: Beginning new tablet copy session on tablet b6a26e01d42b429694641f0ceca11265 from peer bf8ac71b15f740a0a928911360773442 at {username='slave'} at 127.20.207.4:39607: session id = bf8ac71b15f740a0a928911360773442-b6a26e01d42b429694641f0ceca11265
I20240502 08:17:58.772135 27410 tablet_copy_source_session.cc:205] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:58.773098 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet 81e73229c0d145e7bb81761bf4b820dc (table rebalancer_extra_table_0 [id=3600173badac4b5bbac03472cd151621]) successfully deleted
I20240502 08:17:58.774190 31315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a26e01d42b429694641f0ceca11265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.775986 31316 tablet_bootstrap.cc:492] T d00f25f29f5d4db2a2261dd37c0793fa P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:17:58.775954 27112 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet 8c75aa1b58874966994f6cfa7258505a (table rebalancer_dynamic_table [id=07e21a6cd16e4614b0a3130f7a9219e4]). Expected version 1 got 0
I20240502 08:17:58.784089 30289 consensus_queue.cc:237] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:17:58.785799 27650 raft_consensus.cc:1261] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:58.786172 27258 raft_consensus.cc:1261] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:58.786448 31024 consensus_queue.cc:1035] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:58.786705 31266 consensus_queue.cc:1035] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:58.787442 31305 tablet_copy_client.cc:783] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:58.787747 31305 tablet_copy_client.cc:647] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:58.790045 31024 raft_consensus.cc:2932] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:17:58.791184 31305 tablet_copy_client.cc:523] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:58.794140 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 08245cda17c44a77bcde2f864d2045ec with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:58.795365 27258 raft_consensus.cc:2932] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:17:58.795908 31315 tablet_copy_client.cc:783] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:58.796131 31315 tablet_copy_client.cc:647] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:58.795804 27112 catalog_manager.cc:5484] T 08245cda17c44a77bcde2f864d2045ec P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:58.800343 30422 raft_consensus.cc:2932] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:17:58.800860 27650 raft_consensus.cc:2932] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:17:58.806604 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:17:58.808180 30402 tablet_service.cc:1514] Processing DeleteTablet for tablet 08245cda17c44a77bcde2f864d2045ec with delete_type TABLET_DATA_TOMBSTONED (TS 31a1e33dcdbf42f58ea72ce1c0afb0ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56694
I20240502 08:17:58.816785 31320 tablet.cc:1722] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:58.817190 31320 tablet.cc:1620] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:17:58.818989 31325 tablet.cc:1722] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:58.819475 31325 tablet.cc:1620] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:17:58.825225 27112 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=07e21a6cd16e4614b0a3130f7a9219e4] alter complete (version 2)
I20240502 08:17:58.829628 31305 tablet_bootstrap.cc:492] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:58.832798 31316 ts_tablet_manager.cc:1367] T d00f25f29f5d4db2a2261dd37c0793fa P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.000s
I20240502 08:17:58.834551 31110 raft_consensus.cc:482] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:58.834739 31110 raft_consensus.cc:504] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:17:58.835562 31110 leader_election.cc:288] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:17:58.835583 31344 raft_consensus.cc:482] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:58.835717 31344 raft_consensus.cc:504] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:58.836524 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57505255575249e698bf35929619423d" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:17:58.836719 30028 raft_consensus.cc:2454] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:17:58.837095 30093 leader_election.cc:302] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:58.837815 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57505255575249e698bf35929619423d" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:17:58.838037 30290 raft_consensus.cc:2454] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:17:58.838888 31316 raft_consensus.cc:348] T d00f25f29f5d4db2a2261dd37c0793fa P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:58.839444 31327 tablet.cc:1722] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:17:58.840039 31327 tablet.cc:1620] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:17:58.841641 31316 raft_consensus.cc:374] T d00f25f29f5d4db2a2261dd37c0793fa P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:58.842037 31316 raft_consensus.cc:729] T d00f25f29f5d4db2a2261dd37c0793fa P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:17:58.842149 31110 raft_consensus.cc:2781] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:58.842298 31110 raft_consensus.cc:482] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:58.842393 31110 raft_consensus.cc:3037] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:58.842942 31316 consensus_queue.cc:260] T d00f25f29f5d4db2a2261dd37c0793fa P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:58.843266 31315 tablet_copy_client.cc:523] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:58.846866 31316 ts_tablet_manager.cc:1398] T d00f25f29f5d4db2a2261dd37c0793fa P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 08:17:58.850904 31110 raft_consensus.cc:504] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:17:58.851888 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57505255575249e698bf35929619423d" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:17:58.852061 30289 raft_consensus.cc:3037] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:58.852490 31347 tablet_replica.cc:330] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:17:58.852660 31347 raft_consensus.cc:2229] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:58.855324 30289 raft_consensus.cc:2454] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:17:58.855957 30093 leader_election.cc:302] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:58.856247 30981 raft_consensus.cc:2781] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:58.856405 30981 raft_consensus.cc:686] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:17:58.856642 30981 consensus_queue.cc:237] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:17:58.856920 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57505255575249e698bf35929619423d" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:17:58.857159 30028 raft_consensus.cc:3037] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:58.857910 30556 raft_consensus.cc:1050] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 to VOTER
I20240502 08:17:58.860512 30556 consensus_queue.cc:237] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:17:58.860895 30028 raft_consensus.cc:2454] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:17:58.861145 31110 leader_election.cc:288] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:17:58.863873 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dea1ac1af1b45cf9abfd5e5a1d910a1" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:17:58.864148 30289 raft_consensus.cc:2454] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:17:58.864527 29961 leader_election.cc:302] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:17:58.864830 31348 raft_consensus.cc:2781] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:58.865458 31348 raft_consensus.cc:482] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:58.865629 31348 raft_consensus.cc:3037] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:58.865753 31344 leader_election.cc:288] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:17:58.869673 31348 raft_consensus.cc:504] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:58.870817 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dea1ac1af1b45cf9abfd5e5a1d910a1" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:17:58.871035 30289 raft_consensus.cc:3037] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:58.871089 31344 raft_consensus.cc:482] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:58.871223 31344 raft_consensus.cc:504] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:58.872109 30158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00f25f29f5d4db2a2261dd37c0793fa" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:17:58.872305 30158 raft_consensus.cc:2454] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:17:58.872710 29964 leader_election.cc:302] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:58.873286 30289 raft_consensus.cc:2454] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:17:58.874388 29961 leader_election.cc:302] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:17:58.874673 31349 raft_consensus.cc:2781] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:58.874779 31349 raft_consensus.cc:482] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:58.874912 31349 raft_consensus.cc:3037] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:58.877341 31349 raft_consensus.cc:504] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:58.878118 30968 raft_consensus.cc:2781] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:58.878257 30968 raft_consensus.cc:686] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:17:58.878357 30158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00f25f29f5d4db2a2261dd37c0793fa" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:17:58.878528 30158 raft_consensus.cc:3037] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:58.878520 30968 consensus_queue.cc:237] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:58.881055 30158 raft_consensus.cc:2454] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:17:58.882182 31344 leader_election.cc:288] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:58.882556 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dea1ac1af1b45cf9abfd5e5a1d910a1" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
W20240502 08:17:58.883955 29961 leader_election.cc:341] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:58.888720 31349 leader_election.cc:288] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:58.896044 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dea1ac1af1b45cf9abfd5e5a1d910a1" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
W20240502 08:17:58.896834 29961 leader_election.cc:341] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:58.900038 31321 tablet_bootstrap.cc:492] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:17:58.899771 27110 catalog_manager.cc:5484] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:58.900318 31321 ts_tablet_manager.cc:1367] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.231s user 0.008s sys 0.000s
I20240502 08:17:58.901089 31321 raft_consensus.cc:348] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:58.901912 31321 raft_consensus.cc:374] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:58.902094 31321 raft_consensus.cc:729] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:17:58.902406 31321 consensus_queue.cc:260] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:58.905009 31321 ts_tablet_manager.cc:1398] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:17:58.897331 31348 leader_election.cc:288] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:17:58.912889 31305 tablet_bootstrap.cc:492] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:58.913250 29964 leader_election.cc:302] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:17:58.913427 31305 tablet_bootstrap.cc:492] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap complete.
I20240502 08:17:58.913683 31305 ts_tablet_manager.cc:1367] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.084s user 0.011s sys 0.004s
I20240502 08:17:58.913859 31348 raft_consensus.cc:2781] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:58.914127 31348 raft_consensus.cc:686] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:17:58.914531 31348 consensus_queue.cc:237] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:17:58.915885 31315 tablet_bootstrap.cc:492] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:17:58.921965 27110 catalog_manager.cc:5484] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:58.923517 31305 raft_consensus.cc:348] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:17:58.926785 27110 catalog_manager.cc:5484] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:58.927896 31305 raft_consensus.cc:729] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: LEARNER
I20240502 08:17:58.928790 27116 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:17:58.929495 27650 raft_consensus.cc:1261] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:58.930718 27258 raft_consensus.cc:1261] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:58.931051 31305 consensus_queue.cc:260] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:17:58.931664 30289 raft_consensus.cc:1261] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:17:58.934736 31305 ts_tablet_manager.cc:1398] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20240502 08:17:58.934871 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00f25f29f5d4db2a2261dd37c0793fa" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:17:58.935092 27650 raft_consensus.cc:2454] T d00f25f29f5d4db2a2261dd37c0793fa P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:17:58.935705 30180 tablet_copy_service.cc:342] P 84cda20738d2452da166e31fc6da57a6: Request end of tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-ec6c977cac054b758a43d846e821707a received from {username='slave'} at 127.20.207.1:35051
I20240502 08:17:58.935870 30180 tablet_copy_service.cc:433] P 84cda20738d2452da166e31fc6da57a6: ending tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-ec6c977cac054b758a43d846e821707a on tablet ec6c977cac054b758a43d846e821707a with peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60
I20240502 08:17:58.935930 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00f25f29f5d4db2a2261dd37c0793fa" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:17:58.936107 27650 raft_consensus.cc:3037] T d00f25f29f5d4db2a2261dd37c0793fa P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:58.932956 31350 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20240502 08:17:58.940310 31357 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:58.943135 31325 tablet.cc:1722] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:58.943401 31327 tablet.cc:1722] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:58.944772 31357 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:58.951159 27650 raft_consensus.cc:2454] T d00f25f29f5d4db2a2261dd37c0793fa P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:17:58.954146 27112 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=07e21a6cd16e4614b0a3130f7a9219e4] alter complete (version 3)
I20240502 08:17:58.917984 31347 raft_consensus.cc:2258] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:58.949764 31320 tablet.cc:1722] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:17:58.971532 31347 ts_tablet_manager.cc:1875] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:58.973248 27256 raft_consensus.cc:1203] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:17:58.994642 31357 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:17:58.995918 31167 tablet_bootstrap.cc:492] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:58.996369 31167 tablet_bootstrap.cc:492] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap complete.
I20240502 08:17:58.996637 31167 ts_tablet_manager.cc:1367] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.239s user 0.058s sys 0.005s
I20240502 08:17:58.997382 31167 raft_consensus.cc:348] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:17:58.997669 31167 raft_consensus.cc:729] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: LEARNER
I20240502 08:17:58.997886 31167 consensus_queue.cc:260] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:17:58.998510 31167 ts_tablet_manager.cc:1398] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:17:59.000267 27652 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:17:59.007550 27258 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:17:59.010847 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-2dfa509d47e54b0f9ccacc9954981fb6 received from {username='slave'} at 127.20.207.7:45231
I20240502 08:17:59.010963 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-2dfa509d47e54b0f9ccacc9954981fb6 on tablet 2dfa509d47e54b0f9ccacc9954981fb6 with peer 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:17:59.014731 27112 catalog_manager.cc:5484] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:59.020153 27520 consensus_queue.cc:237] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:17:59.022897 27652 raft_consensus.cc:1261] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:59.021323 30289 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:17:59.023711 31350 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:59.024909 31347 ts_tablet_manager.cc:1888] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:17:59.026576 31347 log.cc:1196] T 08245cda17c44a77bcde2f864d2045ec P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/08245cda17c44a77bcde2f864d2045ec
I20240502 08:17:59.027701 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet 08245cda17c44a77bcde2f864d2045ec (table rebalancer_extra_table_2 [id=e14b3e831e834278ac631fa1cb84dced]) successfully deleted
I20240502 08:17:59.036513 31315 tablet_bootstrap.cc:492] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:59.036865 31315 tablet_bootstrap.cc:492] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442: Bootstrap complete.
I20240502 08:17:59.037076 31315 ts_tablet_manager.cc:1367] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.121s user 0.002s sys 0.002s
I20240502 08:17:59.037655 31315 raft_consensus.cc:348] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:17:59.037961 31315 raft_consensus.cc:729] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: LEARNER
I20240502 08:17:59.038239 31315 consensus_queue.cc:260] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:17:59.038921 31315 ts_tablet_manager.cc:1398] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 08:17:59.039582 30288 raft_consensus.cc:1261] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:17:59.039609 27410 tablet_copy_service.cc:342] P 83e171a5e16c4008923f1d43fc5ce7b1: Request end of tablet copy session bf8ac71b15f740a0a928911360773442-b6a26e01d42b429694641f0ceca11265 received from {username='slave'} at 127.20.207.4:39607
I20240502 08:17:59.039784 27410 tablet_copy_service.cc:433] P 83e171a5e16c4008923f1d43fc5ce7b1: ending tablet copy session bf8ac71b15f740a0a928911360773442-b6a26e01d42b429694641f0ceca11265 on tablet b6a26e01d42b429694641f0ceca11265 with peer bf8ac71b15f740a0a928911360773442
I20240502 08:17:59.041369 31359 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:17:59.042380 31359 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:17:59.046130 27652 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:17:59.047770 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c2776f857a5f47e9b8022639dc54757d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:59.050554 27116 catalog_manager.cc:5484] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:59.051214 30288 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:17:59.057004 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet c2776f857a5f47e9b8022639dc54757d with delete_type TABLET_DATA_TOMBSTONED (TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39166
I20240502 08:17:59.061344 27109 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:17:59.061529 27109 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:17:59.065454 27109 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 8c75aa1b58874966994f6cfa7258505a
I20240502 08:17:59.066870 30139 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c75aa1b58874966994f6cfa7258505a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:59 UTC) from {username='slave'} at 127.0.0.1:48808
I20240502 08:17:59.067694 30402 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c75aa1b58874966994f6cfa7258505a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:59 UTC) from {username='slave'} at 127.0.0.1:56694
I20240502 08:17:59.067867 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c75aa1b58874966994f6cfa7258505a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:59 UTC) from {username='slave'} at 127.0.0.1:52730
I20240502 08:17:59.068008 31347 tablet_replica.cc:330] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:17:59.068204 31322 tablet_replica.cc:330] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:17:59.068225 31347 raft_consensus.cc:2229] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:59.068439 31322 raft_consensus.cc:2229] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:59.068553 31347 raft_consensus.cc:2258] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:59.068686 31322 raft_consensus.cc:2258] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:59.069355 31347 ts_tablet_manager.cc:1875] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:59.069792 31322 ts_tablet_manager.cc:1875] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:59.070051 31366 tablet_replica.cc:330] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:17:59.070268 31366 raft_consensus.cc:2229] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:59.070502 31366 raft_consensus.cc:2258] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:59.071528 31366 ts_tablet_manager.cc:1875] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:59.076779 31347 ts_tablet_manager.cc:1888] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:59.076915 31347 log.cc:1196] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/8c75aa1b58874966994f6cfa7258505a
I20240502 08:17:59.078518 31366 ts_tablet_manager.cc:1888] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:17:59.078518 31322 ts_tablet_manager.cc:1888] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:59.078764 31366 log.cc:1196] T c2776f857a5f47e9b8022639dc54757d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/c2776f857a5f47e9b8022639dc54757d
I20240502 08:17:59.080168 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet c2776f857a5f47e9b8022639dc54757d (table rebalance_test_table_5 [id=5c5af31071894d4f803992d5173e0429]) successfully deleted
I20240502 08:17:59.080296 31322 log.cc:1196] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/8c75aa1b58874966994f6cfa7258505a
I20240502 08:17:59.080701 31322 ts_tablet_manager.cc:1909] T 8c75aa1b58874966994f6cfa7258505a P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting consensus metadata
I20240502 08:17:59.082295 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet 8c75aa1b58874966994f6cfa7258505a (table rebalancer_dynamic_table_new_name [id=07e21a6cd16e4614b0a3130f7a9219e4]) successfully deleted
I20240502 08:17:59.082631 31347 ts_tablet_manager.cc:1909] T 8c75aa1b58874966994f6cfa7258505a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting consensus metadata
I20240502 08:17:59.083520 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet 8c75aa1b58874966994f6cfa7258505a (table rebalancer_dynamic_table_new_name [id=07e21a6cd16e4614b0a3130f7a9219e4]) successfully deleted
I20240502 08:17:59.087879 31367 tablet_replica.cc:330] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:17:59.088198 31367 raft_consensus.cc:2229] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:59.088487 31367 raft_consensus.cc:2258] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:59.089334 31367 ts_tablet_manager.cc:1875] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:59.094877 31367 ts_tablet_manager.cc:1888] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:17:59.095032 31367 log.cc:1196] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/8c75aa1b58874966994f6cfa7258505a
I20240502 08:17:59.095561 31367 ts_tablet_manager.cc:1909] T 8c75aa1b58874966994f6cfa7258505a P 84cda20738d2452da166e31fc6da57a6: Deleting consensus metadata
I20240502 08:17:59.096596 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet 8c75aa1b58874966994f6cfa7258505a (table rebalancer_dynamic_table_new_name [id=07e21a6cd16e4614b0a3130f7a9219e4]) successfully deleted
I20240502 08:17:59.109855 27652 raft_consensus.cc:1203] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:17:59.147373 27652 consensus_queue.cc:237] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:17:59.149884 27388 raft_consensus.cc:1261] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:59.150123 27520 raft_consensus.cc:1261] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:59.150635 31291 consensus_queue.cc:1035] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:59.150976 31365 consensus_queue.cc:1035] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:59.153607 31291 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:17:59.154465 27520 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:17:59.155037 27388 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:17:59.161052 27112 catalog_manager.cc:5484] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:17:59.169242 27109 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240502 08:17:59.175469 27584 consensus_peers.cc:485] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 -> Peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Couldn't send request to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33aee3df5eb242ef9cc056d25f6d6a82. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:59.176483 31309 raft_consensus.cc:482] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:59.176666 31309 raft_consensus.cc:504] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:17:59.177320 31309 leader_election.cc:288] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:17:59.178335 30007 tablet_service.cc:1467] Processing CreateTablet for tablet b54f007f0d09495ab64b1bb86b92da94 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1a55c69a6cd64476a588a50861e0ff63]), partition=
I20240502 08:17:59.178544 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dea1ac1af1b45cf9abfd5e5a1d910a1" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:17:59.178846 30289 raft_consensus.cc:2379] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in current term 1: Already voted for candidate 1f0bf4bc3c5146539146b825b048bff3 in this term.
I20240502 08:17:59.178826 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dea1ac1af1b45cf9abfd5e5a1d910a1" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:17:59.179638 30355 leader_election.cc:302] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 1f0bf4bc3c5146539146b825b048bff3, 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:17:59.180060 27370 tablet_service.cc:1467] Processing CreateTablet for tablet b54f007f0d09495ab64b1bb86b92da94 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1a55c69a6cd64476a588a50861e0ff63]), partition=
I20240502 08:17:59.180091 31309 raft_consensus.cc:3037] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:59.180181 27500 tablet_service.cc:1467] Processing CreateTablet for tablet b54f007f0d09495ab64b1bb86b92da94 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1a55c69a6cd64476a588a50861e0ff63]), partition=
I20240502 08:17:59.180413 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54f007f0d09495ab64b1bb86b92da94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:59.180477 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54f007f0d09495ab64b1bb86b92da94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:59.183198 31309 raft_consensus.cc:2726] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:59.183248 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54f007f0d09495ab64b1bb86b92da94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:59.187999 31385 tablet_bootstrap.cc:492] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:59.189364 31385 tablet_bootstrap.cc:654] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.196774 31308 tablet_bootstrap.cc:492] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:59.198314 31308 tablet_bootstrap.cc:654] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.201727 27652 consensus_queue.cc:237] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:17:59.205034 30289 raft_consensus.cc:1261] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:59.207845 31387 tablet_bootstrap.cc:492] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:17:59.208055 30028 raft_consensus.cc:1261] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:59.208917 31288 consensus_queue.cc:1035] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:17:59.209587 31288 consensus_queue.cc:1035] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:17:59.210757 31387 tablet_bootstrap.cc:654] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.212679 31385 tablet_bootstrap.cc:492] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:59.212815 31385 ts_tablet_manager.cc:1367] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.008s
I20240502 08:17:59.214287 31308 tablet_bootstrap.cc:492] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:59.214443 31308 ts_tablet_manager.cc:1367] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.006s
I20240502 08:17:59.214372 31385 raft_consensus.cc:348] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:59.214578 31385 raft_consensus.cc:374] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.214648 31385 raft_consensus.cc:729] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.215147 31308 raft_consensus.cc:348] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:59.215356 31308 raft_consensus.cc:374] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.215426 31308 raft_consensus.cc:729] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.215476 31385 consensus_queue.cc:260] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:59.215605 31365 raft_consensus.cc:2932] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:17:59.216140 31308 consensus_queue.cc:260] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:59.216660 31308 ts_tablet_manager.cc:1398] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:59.217938 31385 ts_tablet_manager.cc:1398] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
W20240502 08:17:59.221326 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84953bd8daba4885be1f5f9ad4149aba. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:59.222198 30028 raft_consensus.cc:2932] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:17:59.222726 27109 catalog_manager.cc:5484] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:59.224615 31387 tablet_bootstrap.cc:492] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:17:59.224741 31387 ts_tablet_manager.cc:1367] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.003s
I20240502 08:17:59.224470 30289 raft_consensus.cc:2932] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:17:59.225824 31387 raft_consensus.cc:348] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:59.226049 31387 raft_consensus.cc:374] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.226140 31387 raft_consensus.cc:729] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.226411 31387 consensus_queue.cc:260] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:59.232622 31387 ts_tablet_manager.cc:1398] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:17:59.257562 27258 consensus_queue.cc:237] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:17:59.267452 27652 raft_consensus.cc:1261] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:59.267590 27520 raft_consensus.cc:1261] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:59.268153 31009 consensus_queue.cc:1035] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:17:59.268819 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3047a6db2c9d438eb5720d848b9ea5bd. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:59.268972 30871 consensus_queue.cc:1035] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:59.270846 31009 raft_consensus.cc:2932] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:17:59.271328 27520 raft_consensus.cc:2932] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:17:59.274976 31110 consensus_queue.cc:1035] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:59.279950 27112 catalog_manager.cc:5484] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:17:59.293279 30981 consensus_queue.cc:1035] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:59.298923 27652 raft_consensus.cc:2932] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:17:59.304924 30968 consensus_queue.cc:1035] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:59.310083 30968 consensus_queue.cc:1035] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:59.326097 27652 consensus_queue.cc:237] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:17:59.328140 27520 raft_consensus.cc:1261] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:59.328373 27258 raft_consensus.cc:1261] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:17:59.328640 27584 consensus_peers.cc:485] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 -> Peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Couldn't send request to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a31709aa6b2b4ff784114fb48cbc8f2a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:59.328820 31365 consensus_queue.cc:1035] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:59.329250 31288 consensus_queue.cc:1035] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:59.331465 31365 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:17:59.332324 27520 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:17:59.332835 27258 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:17:59.337153 27109 catalog_manager.cc:5484] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:59.355297 30028 consensus_queue.cc:237] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:17:59.358038 30422 raft_consensus.cc:1261] T 8bb679bf525a4a2ebafeb73740230569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:17:59.358264 29964 consensus_peers.cc:485] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Couldn't send request to peer bf8ac71b15f740a0a928911360773442. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8bb679bf525a4a2ebafeb73740230569. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:59.358548 30158 raft_consensus.cc:1261] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:59.359086 31348 consensus_queue.cc:1035] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:17:59.359762 31348 consensus_queue.cc:1035] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:17:59.362768 30968 raft_consensus.cc:2932] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:17:59.363415 30158 raft_consensus.cc:2932] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:17:59.367295 30422 raft_consensus.cc:2932] T 8bb679bf525a4a2ebafeb73740230569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:17:59.368144 27116 catalog_manager.cc:5484] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:59.389118 31368 raft_consensus.cc:482] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:59.389294 31368 raft_consensus.cc:504] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:59.389918 31368 leader_election.cc:288] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:59.390518 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54f007f0d09495ab64b1bb86b92da94" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:17:59.390722 27520 raft_consensus.cc:2454] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:59.391651 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54f007f0d09495ab64b1bb86b92da94" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:17:59.391862 30028 raft_consensus.cc:2454] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:17:59.392262 27326 leader_election.cc:302] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:59.392531 31369 raft_consensus.cc:2781] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:59.392602 31369 raft_consensus.cc:482] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:59.392688 31369 raft_consensus.cc:3037] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:59.395900 31369 raft_consensus.cc:504] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:59.396324 27520 consensus_queue.cc:237] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:17:59.397106 27521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54f007f0d09495ab64b1bb86b92da94" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:17:59.397086 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54f007f0d09495ab64b1bb86b92da94" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:17:59.397294 30028 raft_consensus.cc:3037] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:59.397306 27521 raft_consensus.cc:3037] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:59.400089 27521 raft_consensus.cc:2454] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:59.400494 27326 leader_election.cc:302] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:17:59.400789 31368 raft_consensus.cc:2781] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:59.400911 31368 raft_consensus.cc:686] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:17:59.401139 31368 consensus_queue.cc:237] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:17:59.402457 31369 leader_election.cc:288] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:59.402927 30028 raft_consensus.cc:2454] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:17:59.408403 27388 raft_consensus.cc:1261] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:59.409018 31359 consensus_queue.cc:1035] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:59.409875 27109 catalog_manager.cc:5484] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:59.413297 27652 raft_consensus.cc:1261] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:59.414101 31360 consensus_queue.cc:1035] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:59.413275 31350 raft_consensus.cc:2932] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:17:59.413866 27388 raft_consensus.cc:2932] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
W20240502 08:17:59.417105 27453 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 -> Peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Couldn't send request to peer 1f0bf4bc3c5146539146b825b048bff3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82841ff10f24cf7a0cf490be3defe69. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:59.420862 27112 catalog_manager.cc:5484] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:17:59.423036 31349 consensus_queue.cc:1035] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:59.431027 27652 raft_consensus.cc:2932] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:17:59.439209 31349 consensus_queue.cc:1035] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:59.452950 27256 consensus_queue.cc:237] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:17:59.455340 27650 raft_consensus.cc:1261] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:17:59.455581 27191 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15a333c65455473fbdc26d1ec137aebc. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:59.455590 27112 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:59.456653 31009 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:59.457607 27520 raft_consensus.cc:1261] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:59.458562 30871 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:59.460566 30871 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:17:59.461165 27650 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:17:59.467073 27112 catalog_manager.cc:5484] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:17:59.467792 27520 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:17:59.472012 30402 tablet_service.cc:1467] Processing CreateTablet for tablet feb9cd278950480fb63be0be950d190b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=95b099178a3c41f7b5f6e2f30dfe9f42]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:59.472436 30402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feb9cd278950480fb63be0be950d190b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:59.472553 30007 tablet_service.cc:1467] Processing CreateTablet for tablet 41f6a5115030489fac2e1edce67a4b52 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=95b099178a3c41f7b5f6e2f30dfe9f42]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:59.472927 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f6a5115030489fac2e1edce67a4b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:59.478806 27370 tablet_service.cc:1467] Processing CreateTablet for tablet feb9cd278950480fb63be0be950d190b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=95b099178a3c41f7b5f6e2f30dfe9f42]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:59.479348 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feb9cd278950480fb63be0be950d190b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:59.483508 27238 tablet_service.cc:1467] Processing CreateTablet for tablet feb9cd278950480fb63be0be950d190b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=95b099178a3c41f7b5f6e2f30dfe9f42]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:59.483937 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feb9cd278950480fb63be0be950d190b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:59.484853 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 41f6a5115030489fac2e1edce67a4b52 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=95b099178a3c41f7b5f6e2f30dfe9f42]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:59.485277 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f6a5115030489fac2e1edce67a4b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:59.486608 30139 tablet_service.cc:1467] Processing CreateTablet for tablet 41f6a5115030489fac2e1edce67a4b52 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=95b099178a3c41f7b5f6e2f30dfe9f42]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:59.487128 30139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f6a5115030489fac2e1edce67a4b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:59.487638 31308 tablet_bootstrap.cc:492] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:17:59.489346 31308 tablet_bootstrap.cc:654] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.490172 30401 tablet_service.cc:1467] Processing CreateTablet for tablet b1008f7127704fe8b3eb6f9246b0a79c (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=95b099178a3c41f7b5f6e2f30dfe9f42]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:59.490546 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1008f7127704fe8b3eb6f9246b0a79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:59.491102 27368 tablet_service.cc:1467] Processing CreateTablet for tablet b1008f7127704fe8b3eb6f9246b0a79c (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=95b099178a3c41f7b5f6e2f30dfe9f42]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:59.491626 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1008f7127704fe8b3eb6f9246b0a79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:59.498291 30270 tablet_service.cc:1467] Processing CreateTablet for tablet b1008f7127704fe8b3eb6f9246b0a79c (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=95b099178a3c41f7b5f6e2f30dfe9f42]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:59.498699 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1008f7127704fe8b3eb6f9246b0a79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:59.505590 31385 tablet_bootstrap.cc:492] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:17:59.507447 31385 tablet_bootstrap.cc:654] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.490556 31448 tablet_bootstrap.cc:492] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:17:59.513727 31448 tablet_bootstrap.cc:654] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.517807 27520 consensus_queue.cc:237] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:17:59.522291 31368 raft_consensus.cc:1050] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1: attempting to promote NON_VOTER bf8ac71b15f740a0a928911360773442 to VOTER
I20240502 08:17:59.523429 31368 consensus_queue.cc:237] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:17:59.523766 31452 tablet_bootstrap.cc:492] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:59.525717 31452 tablet_bootstrap.cc:654] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.529902 31110 raft_consensus.cc:1050] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6: attempting to promote NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 to VOTER
I20240502 08:17:59.531093 31110 consensus_queue.cc:237] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:17:59.532398 31451 tablet_bootstrap.cc:492] T 41f6a5115030489fac2e1edce67a4b52 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:17:59.534384 31451 tablet_bootstrap.cc:654] T 41f6a5115030489fac2e1edce67a4b52 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.539116 27388 raft_consensus.cc:1261] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:59.539772 31360 consensus_queue.cc:1035] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:17:59.541587 27453 consensus_peers.cc:485] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 -> Peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Couldn't send request to peer 1f0bf4bc3c5146539146b825b048bff3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2b08a6b38f945b1b99f1d8ad1fd0ae0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:59.542567 27256 raft_consensus.cc:1261] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:59.543246 31360 consensus_queue.cc:1035] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:59.548856 30288 raft_consensus.cc:1203] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:59.548799 27109 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:17:59.569434 31450 tablet_bootstrap.cc:492] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:17:59.571882 31450 tablet_bootstrap.cc:654] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.574505 27650 raft_consensus.cc:1261] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:59.575146 31204 consensus_queue.cc:1035] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:17:59.576442 31308 tablet_bootstrap.cc:492] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:17:59.576957 31308 ts_tablet_manager.cc:1367] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.089s user 0.029s sys 0.046s
I20240502 08:17:59.578207 31308 raft_consensus.cc:348] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:59.578646 31308 raft_consensus.cc:374] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.578815 31308 raft_consensus.cc:729] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.579241 31308 consensus_queue.cc:260] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:59.582273 31452 tablet_bootstrap.cc:492] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:59.582528 31452 ts_tablet_manager.cc:1367] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
I20240502 08:17:59.583760 31452 raft_consensus.cc:348] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:59.583994 31452 raft_consensus.cc:374] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.586686 30288 raft_consensus.cc:1261] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:59.587693 30981 consensus_queue.cc:1035] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:17:59.585558 27258 raft_consensus.cc:1261] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:59.587957 31360 raft_consensus.cc:2932] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:17:59.589284 31451 tablet_bootstrap.cc:492] T 41f6a5115030489fac2e1edce67a4b52 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:17:59.589911 30981 consensus_queue.cc:1035] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:17:59.591066 27258 raft_consensus.cc:2932] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:17:59.589473 31451 ts_tablet_manager.cc:1367] T 41f6a5115030489fac2e1edce67a4b52 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.000s
I20240502 08:17:59.596907 31451 raft_consensus.cc:348] T 41f6a5115030489fac2e1edce67a4b52 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:59.584081 31452 raft_consensus.cc:729] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.599144 31452 consensus_queue.cc:260] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:59.601861 31452 ts_tablet_manager.cc:1398] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20240502 08:17:59.604715 27110 catalog_manager.cc:5484] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:17:59.606148 30028 raft_consensus.cc:1261] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:59.606706 31204 consensus_queue.cc:1035] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:17:59.608951 31451 raft_consensus.cc:374] T 41f6a5115030489fac2e1edce67a4b52 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.609047 31451 raft_consensus.cc:729] T 41f6a5115030489fac2e1edce67a4b52 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.610731 31451 consensus_queue.cc:260] T 41f6a5115030489fac2e1edce67a4b52 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:59.611703 31452 tablet_bootstrap.cc:492] T 41f6a5115030489fac2e1edce67a4b52 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:17:59.616441 31450 tablet_bootstrap.cc:492] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:17:59.616591 31450 ts_tablet_manager.cc:1367] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.000s
I20240502 08:17:59.618460 31308 ts_tablet_manager.cc:1398] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20240502 08:17:59.618743 31451 ts_tablet_manager.cc:1398] T 41f6a5115030489fac2e1edce67a4b52 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240502 08:17:59.624073 31452 tablet_bootstrap.cc:654] T 41f6a5115030489fac2e1edce67a4b52 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.617374 31450 raft_consensus.cc:348] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:59.627342 31450 raft_consensus.cc:374] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.627741 31450 raft_consensus.cc:729] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.628240 31450 consensus_queue.cc:260] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:59.629552 31450 ts_tablet_manager.cc:1398] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 08:17:59.640447 31204 raft_consensus.cc:2932] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:17:59.642694 30026 raft_consensus.cc:2932] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:17:59.650483 27388 raft_consensus.cc:2932] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:17:59.651083 27110 catalog_manager.cc:5484] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:17:59.659919 30422 raft_consensus.cc:1261] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:59.661314 27650 raft_consensus.cc:2932] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:17:59.666191 27652 raft_consensus.cc:1261] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:17:59.666195 27388 consensus_queue.cc:237] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:17:59.669538 30026 raft_consensus.cc:1261] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:59.670163 31368 consensus_queue.cc:1035] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:59.670835 27652 raft_consensus.cc:1261] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:59.673291 31110 consensus_queue.cc:1035] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:17:59.672454 31368 consensus_queue.cc:1035] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20240502 08:17:59.693253 30026 raft_consensus.cc:2932] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:17:59.699826 31369 raft_consensus.cc:2932] T b6a26e01d42b429694641f0ceca11265 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:17:59.700596 27110 catalog_manager.cc:5484] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:17:59.706073 30402 tablet_service.cc:1514] Processing DeleteTablet for tablet b6a26e01d42b429694641f0ceca11265 with delete_type TABLET_DATA_TOMBSTONED (TS 31a1e33dcdbf42f58ea72ce1c0afb0ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56694
I20240502 08:17:59.726821 31478 tablet_replica.cc:330] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:17:59.727332 31478 raft_consensus.cc:2229] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:59.727728 31478 raft_consensus.cc:2258] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:59.728646 31478 ts_tablet_manager.cc:1875] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:59.732161 27520 consensus_queue.cc:237] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:17:59.748709 31478 ts_tablet_manager.cc:1888] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:17:59.758133 31478 log.cc:1196] T b6a26e01d42b429694641f0ceca11265 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/b6a26e01d42b429694641f0ceca11265
I20240502 08:17:59.762877 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet b6a26e01d42b429694641f0ceca11265 (table rebalancer_extra_table_4 [id=767cc24e34dd43bc8e4de07a278f8b27]) successfully deleted
I20240502 08:17:59.771538 27651 raft_consensus.cc:2932] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:17:59.781776 31349 raft_consensus.cc:482] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:59.781966 31349 raft_consensus.cc:504] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:59.782678 31349 leader_election.cc:288] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:59.783438 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f6a5115030489fac2e1edce67a4b52" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
W20240502 08:17:59.784098 29961 leader_election.cc:341] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:59.784793 30158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f6a5115030489fac2e1edce67a4b52" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:17:59.785006 30158 raft_consensus.cc:2454] T 41f6a5115030489fac2e1edce67a4b52 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:17:59.785503 29964 leader_election.cc:302] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:17:59.785954 31349 raft_consensus.cc:2781] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:59.786055 31349 raft_consensus.cc:482] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:59.786140 31349 raft_consensus.cc:3037] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:59.788766 31349 raft_consensus.cc:504] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:59.789767 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f6a5115030489fac2e1edce67a4b52" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
W20240502 08:17:59.790473 29961 leader_election.cc:341] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:59.791306 30158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f6a5115030489fac2e1edce67a4b52" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:17:59.791528 30158 raft_consensus.cc:3037] T 41f6a5115030489fac2e1edce67a4b52 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:59.795738 30158 raft_consensus.cc:2454] T 41f6a5115030489fac2e1edce67a4b52 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:17:59.795975 31349 leader_election.cc:288] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:17:59.796517 29964 leader_election.cc:302] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:17:59.797137 31349 raft_consensus.cc:2781] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:59.797513 31349 raft_consensus.cc:686] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:17:59.797870 31349 consensus_queue.cc:237] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:59.804093 31476 ts_tablet_manager.cc:902] T 84953bd8daba4885be1f5f9ad4149aba P 83e171a5e16c4008923f1d43fc5ce7b1: Initiating tablet copy from peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:59.805186 31474 ts_tablet_manager.cc:902] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Initiating tablet copy from peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
W20240502 08:17:59.806596 27323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.5:37815, user_credentials={real_user=slave}} blocked reactor thread for 106995us
I20240502 08:17:59.810777 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b6a26e01d42b429694641f0ceca11265 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:59.824548 30981 raft_consensus.cc:2932] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:17:59.826355 30026 raft_consensus.cc:1261] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:59.827356 31479 consensus_queue.cc:1035] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:59.829308 27256 raft_consensus.cc:2932] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:17:59.830581 27521 raft_consensus.cc:1261] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:59.832199 30289 raft_consensus.cc:2932] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:17:59.848780 31477 consensus_queue.cc:1035] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:59.851645 27110 catalog_manager.cc:5484] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
W20240502 08:17:59.858722 27456 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 -> Peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Couldn't send request to peer 84cda20738d2452da166e31fc6da57a6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb5345b931504cfdbb272189b719efa4. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:59.859019 27256 raft_consensus.cc:1261] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:59.859143 31482 ts_tablet_manager.cc:902] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442: Initiating tablet copy from peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:17:59.859896 31350 consensus_queue.cc:1035] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:17:59.860914 31483 ts_tablet_manager.cc:902] T 3047a6db2c9d438eb5720d848b9ea5bd P 49c5c8f70e494edd9fcda40cc0af6cc4: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:59.865568 27652 raft_consensus.cc:1261] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:17:59.866508 31266 raft_consensus.cc:482] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:59.866665 31266 raft_consensus.cc:504] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:59.867323 31266 leader_election.cc:288] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:59.867388 31491 tablet.cc:1722] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:59.868602 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1008f7127704fe8b3eb6f9246b0a79c" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:17:59.869015 31491 tablet.cc:1620] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1: MemRowSet was empty: no flush needed.
I20240502 08:17:59.869303 31350 consensus_queue.cc:1035] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:17:59.869519 30223 leader_election.cc:341] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:59.875101 31452 tablet_bootstrap.cc:492] T 41f6a5115030489fac2e1edce67a4b52 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:17:59.875308 31452 ts_tablet_manager.cc:1367] T 41f6a5115030489fac2e1edce67a4b52 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.272s user 0.023s sys 0.076s
I20240502 08:17:59.876034 31452 raft_consensus.cc:348] T 41f6a5115030489fac2e1edce67a4b52 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:59.876255 31452 raft_consensus.cc:374] T 41f6a5115030489fac2e1edce67a4b52 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.876336 31452 raft_consensus.cc:729] T 41f6a5115030489fac2e1edce67a4b52 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.876368 31350 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:17:59.876590 31452 consensus_queue.cc:260] T 41f6a5115030489fac2e1edce67a4b52 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:17:59.875753 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=1a55c69a6cd64476a588a50861e0ff63] alter complete (version 1)
I20240502 08:17:59.877535 27256 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:17:59.879846 31452 ts_tablet_manager.cc:1398] T 41f6a5115030489fac2e1edce67a4b52 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240502 08:17:59.881691 31294 raft_consensus.cc:482] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:59.881870 31294 raft_consensus.cc:504] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:59.882750 31294 leader_election.cc:288] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:17:59.889433 27116 catalog_manager.cc:5484] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:17:59.898396 31474 tablet_copy_client.cc:308] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.4:33309
I20240502 08:17:59.897895 31385 tablet_bootstrap.cc:492] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:17:59.899806 31385 ts_tablet_manager.cc:1367] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.394s user 0.011s sys 0.002s
I20240502 08:17:59.899736 27672 tablet_copy_service.cc:140] P bf8ac71b15f740a0a928911360773442: Received BeginTabletCopySession request for tablet 33aee3df5eb242ef9cc056d25f6d6a82 from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad ({username='slave'} at 127.20.207.8:38973)
I20240502 08:17:59.900029 27672 tablet_copy_service.cc:161] P bf8ac71b15f740a0a928911360773442: Beginning new tablet copy session on tablet 33aee3df5eb242ef9cc056d25f6d6a82 from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad at {username='slave'} at 127.20.207.8:38973: session id = 31a1e33dcdbf42f58ea72ce1c0afb0ad-33aee3df5eb242ef9cc056d25f6d6a82
I20240502 08:17:59.900067 31489 ts_tablet_manager.cc:902] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:17:59.901499 31385 raft_consensus.cc:348] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:59.905735 27672 tablet_copy_source_session.cc:205] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:59.913515 27651 raft_consensus.cc:2932] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:17:59.923667 31474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33aee3df5eb242ef9cc056d25f6d6a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:59.928885 31385 raft_consensus.cc:374] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.929790 31385 raft_consensus.cc:729] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.930291 31385 consensus_queue.cc:260] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:59.922230 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1008f7127704fe8b3eb6f9246b0a79c" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:59.899123 31476 tablet_copy_client.cc:308] T 84953bd8daba4885be1f5f9ad4149aba P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.4:33309
I20240502 08:17:59.931516 27672 tablet_copy_service.cc:140] P bf8ac71b15f740a0a928911360773442: Received BeginTabletCopySession request for tablet 84953bd8daba4885be1f5f9ad4149aba from peer 83e171a5e16c4008923f1d43fc5ce7b1 ({username='slave'} at 127.20.207.2:51549)
I20240502 08:17:59.931742 27672 tablet_copy_service.cc:161] P bf8ac71b15f740a0a928911360773442: Beginning new tablet copy session on tablet 84953bd8daba4885be1f5f9ad4149aba from peer 83e171a5e16c4008923f1d43fc5ce7b1 at {username='slave'} at 127.20.207.2:51549: session id = 83e171a5e16c4008923f1d43fc5ce7b1-84953bd8daba4885be1f5f9ad4149aba
I20240502 08:17:59.931732 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feb9cd278950480fb63be0be950d190b" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:17:59.936055 27672 tablet_copy_source_session.cc:205] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:59.938196 31476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84953bd8daba4885be1f5f9ad4149aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:59.943081 31487 ts_tablet_manager.cc:902] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Initiating tablet copy from peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:17:59.945025 27390 raft_consensus.cc:2454] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:17:59.945889 30225 leader_election.cc:302] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1; no voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad
I20240502 08:17:59.946511 31266 raft_consensus.cc:2781] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:59.946612 31266 raft_consensus.cc:482] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:59.946732 31266 raft_consensus.cc:3037] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:59.950654 31266 raft_consensus.cc:504] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:59.951920 31266 leader_election.cc:288] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:17:59.952039 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1008f7127704fe8b3eb6f9246b0a79c" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:17:59.952266 27389 raft_consensus.cc:3037] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:59.953421 31448 tablet_bootstrap.cc:492] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:17:59.953562 31448 ts_tablet_manager.cc:1367] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.466s user 0.055s sys 0.148s
I20240502 08:17:59.954097 27652 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:17:59.955570 31448 raft_consensus.cc:348] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:59.955788 31448 raft_consensus.cc:374] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.955888 31448 raft_consensus.cc:729] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.956111 31448 consensus_queue.cc:260] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:17:59.956701 31448 ts_tablet_manager.cc:1398] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:59.957131 31448 tablet_bootstrap.cc:492] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:17:59.957252 31504 tablet.cc:1722] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:59.958151 31504 tablet.cc:1620] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:17:59.958137 27389 raft_consensus.cc:2454] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:17:59.958751 30225 leader_election.cc:302] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:17:59.959321 31448 tablet_bootstrap.cc:654] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.960404 31266 raft_consensus.cc:2781] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:59.961810 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1008f7127704fe8b3eb6f9246b0a79c" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
W20240502 08:17:59.962739 30223 leader_election.cc:341] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:59.971601 31266 raft_consensus.cc:686] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:17:59.971951 31266 consensus_queue.cc:237] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
W20240502 08:17:59.972430 27193 leader_election.cc:341] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:59.978168 31448 tablet_bootstrap.cc:492] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:17:59.978441 31448 ts_tablet_manager.cc:1367] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.000s
I20240502 08:17:59.979521 31448 raft_consensus.cc:348] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:59.979854 31448 raft_consensus.cc:374] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.979992 31448 raft_consensus.cc:729] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.980368 31448 consensus_queue.cc:260] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:17:59.982707 31448 ts_tablet_manager.cc:1398] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:59.985834 31489 tablet_copy_client.cc:308] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:17:59.986138 31482 tablet_copy_client.cc:308] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.5:37815
I20240502 08:17:59.986984 30048 tablet_copy_service.cc:140] P 1f0bf4bc3c5146539146b825b048bff3: Received BeginTabletCopySession request for tablet 8bb679bf525a4a2ebafeb73740230569 from peer bf8ac71b15f740a0a928911360773442 ({username='slave'} at 127.20.207.4:56663)
I20240502 08:17:59.987241 30048 tablet_copy_service.cc:161] P 1f0bf4bc3c5146539146b825b048bff3: Beginning new tablet copy session on tablet 8bb679bf525a4a2ebafeb73740230569 from peer bf8ac71b15f740a0a928911360773442 at {username='slave'} at 127.20.207.4:56663: session id = bf8ac71b15f740a0a928911360773442-8bb679bf525a4a2ebafeb73740230569
I20240502 08:17:59.986629 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet b82841ff10f24cf7a0cf490be3defe69 from peer 1f0bf4bc3c5146539146b825b048bff3 ({username='slave'} at 127.20.207.5:54501)
I20240502 08:17:59.988399 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet b82841ff10f24cf7a0cf490be3defe69 from peer 1f0bf4bc3c5146539146b825b048bff3 at {username='slave'} at 127.20.207.5:54501: session id = 1f0bf4bc3c5146539146b825b048bff3-b82841ff10f24cf7a0cf490be3defe69
I20240502 08:17:59.993139 27541 tablet_copy_source_session.cc:205] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:59.994529 30048 tablet_copy_source_session.cc:205] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:59.994798 31522 ts_tablet_manager.cc:902] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:17:59.997023 31482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb679bf525a4a2ebafeb73740230569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.001057 27110 catalog_manager.cc:5484] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7). New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:00.005432 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:00.007434 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feb9cd278950480fb63be0be950d190b" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:00.007680 30422 raft_consensus.cc:2454] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:18:00.008423 27191 leader_election.cc:302] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:00.008806 31524 raft_consensus.cc:2781] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:00.008917 31524 raft_consensus.cc:482] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:00.009037 31524 raft_consensus.cc:3037] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.011377 31489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82841ff10f24cf7a0cf490be3defe69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.014953 31524 raft_consensus.cc:504] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:00.016146 31524 leader_election.cc:288] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:00.016085 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feb9cd278950480fb63be0be950d190b" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
W20240502 08:18:00.016590 27193 leader_election.cc:341] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:00.016997 27109 catalog_manager.cc:5576] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469) does not have the latest schema for tablet b54f007f0d09495ab64b1bb86b92da94 (table rebalancer_dynamic_table [id=1a55c69a6cd64476a588a50861e0ff63]). Expected version 2 got 0
I20240502 08:18:00.018414 31482 tablet_copy_client.cc:783] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:00.018782 31482 tablet_copy_client.cc:647] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:00.019943 27116 catalog_manager.cc:5576] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291) does not have the latest schema for tablet b54f007f0d09495ab64b1bb86b92da94 (table rebalancer_dynamic_table [id=1a55c69a6cd64476a588a50861e0ff63]). Expected version 2 got 1
I20240502 08:18:00.020906 31474 tablet_copy_client.cc:783] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:00.021315 31474 tablet_copy_client.cc:647] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:00.022280 31491 tablet.cc:1722] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:00.022722 31491 tablet.cc:1620] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1: MemRowSet was empty: no flush needed.
I20240502 08:18:00.024281 31504 tablet.cc:1722] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:00.024767 31504 tablet.cc:1620] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:00.027295 31385 ts_tablet_manager.cc:1398] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.127s user 0.003s sys 0.000s
I20240502 08:18:00.030591 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feb9cd278950480fb63be0be950d190b" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:00.030879 30422 raft_consensus.cc:3037] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.031216 31527 tablet.cc:1722] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:00.033963 30422 raft_consensus.cc:2454] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:18:00.036725 31482 tablet_copy_client.cc:523] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:00.036753 31474 tablet_copy_client.cc:523] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:00.037003 27191 leader_election.cc:302] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:00.040768 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=1a55c69a6cd64476a588a50861e0ff63] alter complete (version 2)
I20240502 08:18:00.040870 30871 raft_consensus.cc:2781] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:00.041018 30871 raft_consensus.cc:686] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:18:00.041309 30871 consensus_queue.cc:237] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:00.042992 31009 raft_consensus.cc:1050] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 to VOTER
I20240502 08:18:00.043838 31527 tablet.cc:1620] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59: MemRowSet was empty: no flush needed.
I20240502 08:18:00.047536 31474 tablet_bootstrap.cc:492] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:00.049278 31009 consensus_queue.cc:237] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:00.049719 31385 tablet_bootstrap.cc:492] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:00.051569 31385 tablet_bootstrap.cc:654] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.068511 30288 raft_consensus.cc:1261] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:00.069295 31294 consensus_queue.cc:1035] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:00.076138 27116 catalog_manager.cc:5576] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469) does not have the latest schema for tablet b54f007f0d09495ab64b1bb86b92da94 (table rebalancer_dynamic_table [id=1a55c69a6cd64476a588a50861e0ff63]). Expected version 2 got 0
I20240502 08:18:00.077090 27389 raft_consensus.cc:1261] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:00.077620 27520 raft_consensus.cc:1261] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:00.080761 31294 consensus_queue.cc:1035] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:00.080720 27110 catalog_manager.cc:5484] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:00.081284 31294 consensus_queue.cc:1035] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:00.091354 31527 tablet.cc:1722] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:00.091845 31527 tablet.cc:1620] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59: MemRowSet was empty: no flush needed.
I20240502 08:18:00.091986 27109 catalog_manager.cc:5576] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469) does not have the latest schema for tablet b54f007f0d09495ab64b1bb86b92da94 (table rebalancer_dynamic_table [id=1a55c69a6cd64476a588a50861e0ff63]). Expected version 2 got 1
I20240502 08:18:00.097579 31487 tablet_copy_client.cc:308] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.4:33309
I20240502 08:18:00.097653 31482 tablet_bootstrap.cc:492] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:00.098384 27672 tablet_copy_service.cc:140] P bf8ac71b15f740a0a928911360773442: Received BeginTabletCopySession request for tablet a31709aa6b2b4ff784114fb48cbc8f2a from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad ({username='slave'} at 127.20.207.8:38973)
I20240502 08:18:00.098655 27672 tablet_copy_service.cc:161] P bf8ac71b15f740a0a928911360773442: Beginning new tablet copy session on tablet a31709aa6b2b4ff784114fb48cbc8f2a from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad at {username='slave'} at 127.20.207.8:38973: session id = 31a1e33dcdbf42f58ea72ce1c0afb0ad-a31709aa6b2b4ff784114fb48cbc8f2a
I20240502 08:18:00.101698 31483 tablet_copy_client.cc:308] T 3047a6db2c9d438eb5720d848b9ea5bd P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:18:00.102484 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet 3047a6db2c9d438eb5720d848b9ea5bd from peer 49c5c8f70e494edd9fcda40cc0af6cc4 ({username='slave'} at 127.20.207.7:45231)
I20240502 08:18:00.102648 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet 3047a6db2c9d438eb5720d848b9ea5bd from peer 49c5c8f70e494edd9fcda40cc0af6cc4 at {username='slave'} at 127.20.207.7:45231: session id = 49c5c8f70e494edd9fcda40cc0af6cc4-3047a6db2c9d438eb5720d848b9ea5bd
I20240502 08:18:00.107286 31385 tablet_bootstrap.cc:492] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:18:00.107489 31385 ts_tablet_manager.cc:1367] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.080s user 0.004s sys 0.000s
I20240502 08:18:00.108739 31385 raft_consensus.cc:348] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:00.109088 31385 raft_consensus.cc:374] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.109207 31385 raft_consensus.cc:729] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.109447 31385 consensus_queue.cc:260] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:00.110105 27672 tablet_copy_source_session.cc:205] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:00.110735 27278 tablet_copy_source_session.cc:205] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:00.112773 31483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3047a6db2c9d438eb5720d848b9ea5bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.113044 31487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a31709aa6b2b4ff784114fb48cbc8f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.119429 31385 ts_tablet_manager.cc:1398] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:00.121626 31544 ts_tablet_manager.cc:902] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:00.123616 31476 tablet_copy_client.cc:783] T 84953bd8daba4885be1f5f9ad4149aba P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:00.123924 31476 tablet_copy_client.cc:647] T 84953bd8daba4885be1f5f9ad4149aba P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:00.125406 31476 tablet_copy_client.cc:523] T 84953bd8daba4885be1f5f9ad4149aba P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 08:18:00.127652 27193 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.2:46291, user_credentials={real_user=slave}} blocked reactor thread for 48581.7us
I20240502 08:18:00.128959 31544 tablet_copy_client.cc:308] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:00.129621 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet f2b08a6b38f945b1b99f1d8ad1fd0ae0 from peer 1f0bf4bc3c5146539146b825b048bff3 ({username='slave'} at 127.20.207.5:54501)
I20240502 08:18:00.129879 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet f2b08a6b38f945b1b99f1d8ad1fd0ae0 from peer 1f0bf4bc3c5146539146b825b048bff3 at {username='slave'} at 127.20.207.5:54501: session id = 1f0bf4bc3c5146539146b825b048bff3-f2b08a6b38f945b1b99f1d8ad1fd0ae0
I20240502 08:18:00.131925 31476 tablet_bootstrap.cc:492] T 84953bd8daba4885be1f5f9ad4149aba P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:00.133391 27541 tablet_copy_source_session.cc:205] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:00.134768 31294 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:00.139168 31522 tablet_copy_client.cc:308] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:18:00.139606 27521 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:00.141712 31482 tablet_bootstrap.cc:492] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:00.141865 31489 tablet_copy_client.cc:783] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:00.142185 31489 tablet_copy_client.cc:647] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:00.142231 31482 tablet_bootstrap.cc:492] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442: Bootstrap complete.
I20240502 08:18:00.142470 31482 ts_tablet_manager.cc:1367] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.002s
I20240502 08:18:00.143744 31482 raft_consensus.cc:348] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:00.143978 31482 raft_consensus.cc:729] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: LEARNER
I20240502 08:18:00.143913 31489 tablet_copy_client.cc:523] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:00.144207 31482 consensus_queue.cc:260] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:00.144510 31483 tablet_copy_client.cc:783] T 3047a6db2c9d438eb5720d848b9ea5bd P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:00.144732 31483 tablet_copy_client.cc:647] T 3047a6db2c9d438eb5720d848b9ea5bd P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:00.146941 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet 15a333c65455473fbdc26d1ec137aebc from peer 49c5c8f70e494edd9fcda40cc0af6cc4 ({username='slave'} at 127.20.207.7:45231)
I20240502 08:18:00.146622 30288 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:00.147119 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet 15a333c65455473fbdc26d1ec137aebc from peer 49c5c8f70e494edd9fcda40cc0af6cc4 at {username='slave'} at 127.20.207.7:45231: session id = 49c5c8f70e494edd9fcda40cc0af6cc4-15a333c65455473fbdc26d1ec137aebc
I20240502 08:18:00.147827 27109 catalog_manager.cc:5484] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:00.150108 27278 tablet_copy_source_session.cc:205] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:00.152349 31522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a333c65455473fbdc26d1ec137aebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.157518 31544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b08a6b38f945b1b99f1d8ad1fd0ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.159911 31483 tablet_copy_client.cc:523] T 3047a6db2c9d438eb5720d848b9ea5bd P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:00.163859 31483 tablet_bootstrap.cc:492] T 3047a6db2c9d438eb5720d848b9ea5bd P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:00.166348 31482 ts_tablet_manager.cc:1398] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.024s user 0.000s sys 0.002s
I20240502 08:18:00.167354 30048 tablet_copy_service.cc:342] P 1f0bf4bc3c5146539146b825b048bff3: Request end of tablet copy session bf8ac71b15f740a0a928911360773442-8bb679bf525a4a2ebafeb73740230569 received from {username='slave'} at 127.20.207.4:56663
I20240502 08:18:00.167565 30048 tablet_copy_service.cc:433] P 1f0bf4bc3c5146539146b825b048bff3: ending tablet copy session bf8ac71b15f740a0a928911360773442-8bb679bf525a4a2ebafeb73740230569 on tablet 8bb679bf525a4a2ebafeb73740230569 with peer bf8ac71b15f740a0a928911360773442
I20240502 08:18:00.169078 27112 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:00.171943 31522 tablet_copy_client.cc:783] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:00.172168 31522 tablet_copy_client.cc:647] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:00.173645 31522 tablet_copy_client.cc:523] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:00.173264 27389 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:00.181483 31491 tablet.cc:1722] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:00.187130 31538 tablet.cc:1722] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:00.187757 31504 tablet.cc:1722] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:00.191002 31522 tablet_bootstrap.cc:492] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:00.193382 31476 tablet_bootstrap.cc:492] T 84953bd8daba4885be1f5f9ad4149aba P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:00.193809 31476 tablet_bootstrap.cc:492] T 84953bd8daba4885be1f5f9ad4149aba P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap complete.
I20240502 08:18:00.199054 30288 raft_consensus.cc:3037] T 41f6a5115030489fac2e1edce67a4b52 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.203382 31349 consensus_queue.cc:1035] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:00.204548 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=1a55c69a6cd64476a588a50861e0ff63] alter complete (version 3)
I20240502 08:18:00.207681 31476 ts_tablet_manager.cc:1367] T 84953bd8daba4885be1f5f9ad4149aba P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.076s user 0.016s sys 0.002s
I20240502 08:18:00.210364 31476 raft_consensus.cc:348] T 84953bd8daba4885be1f5f9ad4149aba P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:00.210714 31476 raft_consensus.cc:729] T 84953bd8daba4885be1f5f9ad4149aba P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: LEARNER
I20240502 08:18:00.211043 31476 consensus_queue.cc:260] T 84953bd8daba4885be1f5f9ad4149aba P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:00.210795 27109 catalog_manager.cc:5484] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:00.211867 31476 ts_tablet_manager.cc:1398] T 84953bd8daba4885be1f5f9ad4149aba P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:00.211923 31487 tablet_copy_client.cc:783] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:00.212181 31487 tablet_copy_client.cc:647] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:00.212662 27672 tablet_copy_service.cc:342] P bf8ac71b15f740a0a928911360773442: Request end of tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-84953bd8daba4885be1f5f9ad4149aba received from {username='slave'} at 127.20.207.2:51549
I20240502 08:18:00.212817 27672 tablet_copy_service.cc:433] P bf8ac71b15f740a0a928911360773442: ending tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-84953bd8daba4885be1f5f9ad4149aba on tablet 84953bd8daba4885be1f5f9ad4149aba with peer 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:00.213266 31349 consensus_queue.cc:1035] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:00.215423 31544 tablet_copy_client.cc:783] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:00.215682 31544 tablet_copy_client.cc:647] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:00.215917 31479 raft_consensus.cc:482] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:00.216069 31479 raft_consensus.cc:504] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:00.216684 31479 leader_election.cc:288] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:00.218607 27256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feb9cd278950480fb63be0be950d190b" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:00.223268 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feb9cd278950480fb63be0be950d190b" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:00.223528 30422 raft_consensus.cc:2379] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83e171a5e16c4008923f1d43fc5ce7b1 in current term 1: Already voted for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in this term.
I20240502 08:18:00.223958 27323 leader_election.cc:302] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1; no voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 31a1e33dcdbf42f58ea72ce1c0afb0ad
I20240502 08:18:00.224448 31479 raft_consensus.cc:3037] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.224973 31544 tablet_copy_client.cc:523] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:00.227298 31479 raft_consensus.cc:2726] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:00.229681 31489 tablet_bootstrap.cc:492] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:00.237401 31544 tablet_bootstrap.cc:492] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:00.237600 31487 tablet_copy_client.cc:523] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:00.245697 27652 raft_consensus.cc:1203] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:00.255587 31487 tablet_bootstrap.cc:492] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:00.283680 27256 consensus_queue.cc:237] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:00.286819 30288 raft_consensus.cc:1261] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:00.287451 31539 consensus_queue.cc:1035] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:00.289806 31539 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:00.294535 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2dfa509d47e54b0f9ccacc9954981fb6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:00.295329 30288 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:00.304165 27110 catalog_manager.cc:5484] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:00.307992 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dfa509d47e54b0f9ccacc9954981fb6 with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:00.309789 27389 raft_consensus.cc:1261] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:00.312129 31540 consensus_queue.cc:1035] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:00.314893 27389 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:00.338012 27116 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:00.338249 27116 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:00.341619 27116 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet b54f007f0d09495ab64b1bb86b92da94
I20240502 08:18:00.342905 30007 tablet_service.cc:1514] Processing DeleteTablet for tablet b54f007f0d09495ab64b1bb86b92da94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:00 UTC) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:00.343034 27370 tablet_service.cc:1514] Processing DeleteTablet for tablet b54f007f0d09495ab64b1bb86b92da94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:00 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:18:00.353787 31566 tablet_replica.cc:330] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:00.354508 31309 raft_consensus.cc:482] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:00.354615 31566 raft_consensus.cc:2229] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:00.354684 31309 raft_consensus.cc:504] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:00.354947 31566 raft_consensus.cc:2258] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:00.355347 31309 leader_election.cc:288] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:00.355912 31566 ts_tablet_manager.cc:1875] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:00.356441 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1008f7127704fe8b3eb6f9246b0a79c" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:00.358126 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1008f7127704fe8b3eb6f9246b0a79c" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:18:00.358407 27390 raft_consensus.cc:2379] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in current term 1: Already voted for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in this term.
I20240502 08:18:00.358929 30357 leader_election.cc:302] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:00.359653 31309 raft_consensus.cc:3037] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.363941 31309 raft_consensus.cc:2726] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:00.365797 31570 tablet_replica.cc:330] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:18:00.366122 31570 raft_consensus.cc:2229] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:00.366550 31570 raft_consensus.cc:2258] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:00.369515 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet b54f007f0d09495ab64b1bb86b92da94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:00 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:00.370472 31570 ts_tablet_manager.cc:1875] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:00.371975 31572 ts_tablet_manager.cc:902] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:00.374769 31569 tablet_replica.cc:330] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:00.375208 31569 raft_consensus.cc:2229] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:00.375473 31569 raft_consensus.cc:2258] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:00.376480 31569 ts_tablet_manager.cc:1875] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:00.386152 31522 tablet_bootstrap.cc:492] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:00.386607 31522 tablet_bootstrap.cc:492] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap complete.
I20240502 08:18:00.386893 31522 ts_tablet_manager.cc:1367] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.196s user 0.042s sys 0.008s
I20240502 08:18:00.387676 31522 raft_consensus.cc:348] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:00.387933 31522 raft_consensus.cc:729] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: LEARNER
I20240502 08:18:00.388175 31522 consensus_queue.cc:260] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:00.395999 31572 tablet_copy_client.cc:308] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:00.396721 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet eb5345b931504cfdbb272189b719efa4 from peer 84cda20738d2452da166e31fc6da57a6 ({username='slave'} at 127.20.207.6:40325)
I20240502 08:18:00.396893 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet eb5345b931504cfdbb272189b719efa4 from peer 84cda20738d2452da166e31fc6da57a6 at {username='slave'} at 127.20.207.6:40325: session id = 84cda20738d2452da166e31fc6da57a6-eb5345b931504cfdbb272189b719efa4
I20240502 08:18:00.398375 31483 tablet_bootstrap.cc:492] T 3047a6db2c9d438eb5720d848b9ea5bd P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:00.398803 31483 tablet_bootstrap.cc:492] T 3047a6db2c9d438eb5720d848b9ea5bd P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap complete.
I20240502 08:18:00.399050 31483 ts_tablet_manager.cc:1367] T 3047a6db2c9d438eb5720d848b9ea5bd P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.235s user 0.053s sys 0.000s
I20240502 08:18:00.399893 27541 tablet_copy_source_session.cc:205] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:00.400301 31570 ts_tablet_manager.cc:1888] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:00.401345 31572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5345b931504cfdbb272189b719efa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.404461 31569 ts_tablet_manager.cc:1888] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:00.405161 31570 log.cc:1196] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/b54f007f0d09495ab64b1bb86b92da94
I20240502 08:18:00.405678 31570 ts_tablet_manager.cc:1909] T b54f007f0d09495ab64b1bb86b92da94 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:18:00.405450 31569 log.cc:1196] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/b54f007f0d09495ab64b1bb86b92da94
I20240502 08:18:00.406606 31569 ts_tablet_manager.cc:1909] T b54f007f0d09495ab64b1bb86b92da94 P 1f0bf4bc3c5146539146b825b048bff3: Deleting consensus metadata
I20240502 08:18:00.408437 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet b54f007f0d09495ab64b1bb86b92da94 (table rebalancer_dynamic_table_new_name [id=1a55c69a6cd64476a588a50861e0ff63]) successfully deleted
I20240502 08:18:00.410943 31566 ts_tablet_manager.cc:1888] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:00.411053 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet b54f007f0d09495ab64b1bb86b92da94 (table rebalancer_dynamic_table_new_name [id=1a55c69a6cd64476a588a50861e0ff63]) successfully deleted
I20240502 08:18:00.416636 31483 raft_consensus.cc:348] T 3047a6db2c9d438eb5720d848b9ea5bd P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:00.416947 31483 raft_consensus.cc:729] T 3047a6db2c9d438eb5720d848b9ea5bd P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: LEARNER
I20240502 08:18:00.417160 31483 consensus_queue.cc:260] T 3047a6db2c9d438eb5720d848b9ea5bd P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:00.418748 31483 ts_tablet_manager.cc:1398] T 3047a6db2c9d438eb5720d848b9ea5bd P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20240502 08:18:00.419391 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-3047a6db2c9d438eb5720d848b9ea5bd received from {username='slave'} at 127.20.207.7:45231
I20240502 08:18:00.419524 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-3047a6db2c9d438eb5720d848b9ea5bd on tablet 3047a6db2c9d438eb5720d848b9ea5bd with peer 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:00.421344 31522 ts_tablet_manager.cc:1398] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20240502 08:18:00.422984 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-15a333c65455473fbdc26d1ec137aebc received from {username='slave'} at 127.20.207.7:45231
I20240502 08:18:00.423100 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-15a333c65455473fbdc26d1ec137aebc on tablet 15a333c65455473fbdc26d1ec137aebc with peer 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:00.424854 31566 log.cc:1196] T 2dfa509d47e54b0f9ccacc9954981fb6 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/2dfa509d47e54b0f9ccacc9954981fb6
I20240502 08:18:00.428323 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 2dfa509d47e54b0f9ccacc9954981fb6 (table rebalance_test_table_19 [id=7f9af7f2cb49474eb721698045f06a63]) successfully deleted
I20240502 08:18:00.431175 31566 tablet_replica.cc:330] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:00.432971 31474 tablet_bootstrap.cc:492] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:00.433450 31474 tablet_bootstrap.cc:492] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap complete.
I20240502 08:18:00.433688 31566 raft_consensus.cc:2229] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:00.433710 31474 ts_tablet_manager.cc:1367] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.388s user 0.054s sys 0.022s
I20240502 08:18:00.434175 31566 raft_consensus.cc:2258] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:00.434656 31474 raft_consensus.cc:348] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:00.434927 31474 raft_consensus.cc:729] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: LEARNER
I20240502 08:18:00.435184 31474 consensus_queue.cc:260] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:00.436219 30288 raft_consensus.cc:1203] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:00.441172 31572 tablet_copy_client.cc:783] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:00.441277 31566 ts_tablet_manager.cc:1875] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:00.441885 31572 tablet_copy_client.cc:647] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:00.446556 31572 tablet_copy_client.cc:523] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:00.452556 31266 consensus_queue.cc:1035] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:00.460501 31572 tablet_bootstrap.cc:492] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:00.462671 31474 ts_tablet_manager.cc:1398] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20240502 08:18:00.463418 27671 tablet_copy_service.cc:342] P bf8ac71b15f740a0a928911360773442: Request end of tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-33aee3df5eb242ef9cc056d25f6d6a82 received from {username='slave'} at 127.20.207.8:38973
I20240502 08:18:00.463579 27671 tablet_copy_service.cc:433] P bf8ac71b15f740a0a928911360773442: ending tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-33aee3df5eb242ef9cc056d25f6d6a82 on tablet 33aee3df5eb242ef9cc056d25f6d6a82 with peer 31a1e33dcdbf42f58ea72ce1c0afb0ad
I20240502 08:18:00.471282 27109 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:00.479929 31519 consensus_queue.cc:1035] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:00.482758 30007 tablet_service.cc:1467] Processing CreateTablet for tablet c4f8db1d11ec4e59a8eaa8b4b80c1024 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a8189a946a094e6394d4cf6e22461d0d]), partition=
I20240502 08:18:00.483289 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f8db1d11ec4e59a8eaa8b4b80c1024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.483374 30270 tablet_service.cc:1467] Processing CreateTablet for tablet c4f8db1d11ec4e59a8eaa8b4b80c1024 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a8189a946a094e6394d4cf6e22461d0d]), partition=
I20240502 08:18:00.483676 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f8db1d11ec4e59a8eaa8b4b80c1024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.490125 31487 tablet_bootstrap.cc:492] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:00.490216 30402 tablet_service.cc:1467] Processing CreateTablet for tablet c4f8db1d11ec4e59a8eaa8b4b80c1024 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a8189a946a094e6394d4cf6e22461d0d]), partition=
I20240502 08:18:00.490563 31487 tablet_bootstrap.cc:492] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap complete.
I20240502 08:18:00.490628 30402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f8db1d11ec4e59a8eaa8b4b80c1024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.490819 31487 ts_tablet_manager.cc:1367] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.235s user 0.052s sys 0.003s
I20240502 08:18:00.491636 31487 raft_consensus.cc:348] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:00.491919 31487 raft_consensus.cc:729] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: LEARNER
I20240502 08:18:00.492159 31487 consensus_queue.cc:260] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:00.492825 31487 ts_tablet_manager.cc:1398] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:00.493546 27671 tablet_copy_service.cc:342] P bf8ac71b15f740a0a928911360773442: Request end of tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-a31709aa6b2b4ff784114fb48cbc8f2a received from {username='slave'} at 127.20.207.8:38973
I20240502 08:18:00.493659 27671 tablet_copy_service.cc:433] P bf8ac71b15f740a0a928911360773442: ending tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-a31709aa6b2b4ff784114fb48cbc8f2a on tablet a31709aa6b2b4ff784114fb48cbc8f2a with peer 31a1e33dcdbf42f58ea72ce1c0afb0ad
I20240502 08:18:00.495333 30871 consensus_queue.cc:1035] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:00.495877 31544 tablet_bootstrap.cc:492] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:00.496285 31544 tablet_bootstrap.cc:492] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap complete.
I20240502 08:18:00.496562 31544 ts_tablet_manager.cc:1367] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.259s user 0.045s sys 0.006s
I20240502 08:18:00.497349 31544 raft_consensus.cc:348] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:00.497603 31544 raft_consensus.cc:729] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: LEARNER
I20240502 08:18:00.497809 31544 consensus_queue.cc:260] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:00.498369 31587 tablet_bootstrap.cc:492] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:00.500886 31587 tablet_bootstrap.cc:654] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.505945 31544 ts_tablet_manager.cc:1398] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240502 08:18:00.509550 31587 tablet_bootstrap.cc:492] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:00.509703 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 1f0bf4bc3c5146539146b825b048bff3-f2b08a6b38f945b1b99f1d8ad1fd0ae0 received from {username='slave'} at 127.20.207.5:54501
I20240502 08:18:00.509754 31587 ts_tablet_manager.cc:1367] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.007s
I20240502 08:18:00.509848 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 1f0bf4bc3c5146539146b825b048bff3-f2b08a6b38f945b1b99f1d8ad1fd0ae0 on tablet f2b08a6b38f945b1b99f1d8ad1fd0ae0 with peer 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:18:00.511469 31587 raft_consensus.cc:348] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:00.511782 31587 raft_consensus.cc:374] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.511919 31587 raft_consensus.cc:729] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.512226 31587 consensus_queue.cc:260] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:00.513195 31587 ts_tablet_manager.cc:1398] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:00.514096 31589 tablet_bootstrap.cc:492] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:00.515110 31590 tablet_bootstrap.cc:492] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:00.515794 31589 tablet_bootstrap.cc:654] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.523646 31589 tablet_bootstrap.cc:492] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:00.523854 31589 ts_tablet_manager.cc:1367] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240502 08:18:00.524577 31566 ts_tablet_manager.cc:1888] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:00.524729 31566 log.cc:1196] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/b54f007f0d09495ab64b1bb86b92da94
I20240502 08:18:00.524713 31589 raft_consensus.cc:348] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:00.525018 31589 raft_consensus.cc:374] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.525169 31566 ts_tablet_manager.cc:1909] T b54f007f0d09495ab64b1bb86b92da94 P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:18:00.525156 31589 raft_consensus.cc:729] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.525494 31589 consensus_queue.cc:260] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:00.526171 31589 ts_tablet_manager.cc:1398] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:00.526232 31590 tablet_bootstrap.cc:654] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.536140 31590 tablet_bootstrap.cc:492] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:00.536350 31590 ts_tablet_manager.cc:1367] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20240502 08:18:00.537171 31590 raft_consensus.cc:348] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:00.537438 31590 raft_consensus.cc:374] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.537544 31590 raft_consensus.cc:729] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.537811 31590 consensus_queue.cc:260] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:00.541247 31590 ts_tablet_manager.cc:1398] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 08:18:00.555850 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet b54f007f0d09495ab64b1bb86b92da94 (table rebalancer_dynamic_table_new_name [id=1a55c69a6cd64476a588a50861e0ff63]) successfully deleted
I20240502 08:18:00.562541 31540 consensus_queue.cc:1035] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:00.578963 31489 tablet_bootstrap.cc:492] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:00.579490 31489 tablet_bootstrap.cc:492] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap complete.
I20240502 08:18:00.579856 31489 ts_tablet_manager.cc:1367] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.350s user 0.047s sys 0.004s
I20240502 08:18:00.580684 31489 raft_consensus.cc:348] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:00.580997 31489 raft_consensus.cc:729] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: LEARNER
I20240502 08:18:00.581244 31489 consensus_queue.cc:260] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:00.582708 31489 ts_tablet_manager.cc:1398] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:00.586506 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 1f0bf4bc3c5146539146b825b048bff3-b82841ff10f24cf7a0cf490be3defe69 received from {username='slave'} at 127.20.207.5:54501
I20240502 08:18:00.586678 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 1f0bf4bc3c5146539146b825b048bff3-b82841ff10f24cf7a0cf490be3defe69 on tablet b82841ff10f24cf7a0cf490be3defe69 with peer 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:18:00.624157 31572 tablet_bootstrap.cc:492] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:00.624697 31572 tablet_bootstrap.cc:492] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6: Bootstrap complete.
I20240502 08:18:00.625006 31572 ts_tablet_manager.cc:1367] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.165s user 0.044s sys 0.003s
I20240502 08:18:00.625710 31572 raft_consensus.cc:348] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:00.626066 31572 raft_consensus.cc:729] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: LEARNER
I20240502 08:18:00.628031 31572 consensus_queue.cc:260] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:00.629719 31349 raft_consensus.cc:482] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:00.629863 31349 raft_consensus.cc:504] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:00.630578 31349 leader_election.cc:288] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:00.631361 31572 ts_tablet_manager.cc:1398] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:18:00.631641 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f8db1d11ec4e59a8eaa8b4b80c1024" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:00.631858 30421 raft_consensus.cc:2454] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:00.632097 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 84cda20738d2452da166e31fc6da57a6-eb5345b931504cfdbb272189b719efa4 received from {username='slave'} at 127.20.207.6:40325
I20240502 08:18:00.632256 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 84cda20738d2452da166e31fc6da57a6-eb5345b931504cfdbb272189b719efa4 on tablet eb5345b931504cfdbb272189b719efa4 with peer 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:00.632315 29961 leader_election.cc:302] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters:
I20240502 08:18:00.634294 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f8db1d11ec4e59a8eaa8b4b80c1024" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:00.634522 30288 raft_consensus.cc:2454] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:00.635238 31349 raft_consensus.cc:2781] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:00.635353 31349 raft_consensus.cc:482] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:00.635435 31349 raft_consensus.cc:3037] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.639645 30028 raft_consensus.cc:1203] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:00.643357 31349 raft_consensus.cc:504] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:00.644332 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f8db1d11ec4e59a8eaa8b4b80c1024" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:00.644541 30288 raft_consensus.cc:3037] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.646317 31349 leader_election.cc:288] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:00.647979 30288 raft_consensus.cc:2454] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:00.648123 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f8db1d11ec4e59a8eaa8b4b80c1024" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:00.648344 30421 raft_consensus.cc:3037] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.648639 29961 leader_election.cc:302] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:18:00.648923 31349 raft_consensus.cc:2781] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:00.649032 31349 raft_consensus.cc:686] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:18:00.649288 31349 consensus_queue.cc:237] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:00.651170 30421 raft_consensus.cc:2454] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:00.660959 27109 catalog_manager.cc:5484] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:00.663417 27389 raft_consensus.cc:1203] T 84953bd8daba4885be1f5f9ad4149aba P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:00.681926 30288 raft_consensus.cc:1203] T 3047a6db2c9d438eb5720d848b9ea5bd P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:00.708734 31349 raft_consensus.cc:1050] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3: attempting to promote NON_VOTER bf8ac71b15f740a0a928911360773442 to VOTER
I20240502 08:18:00.709725 31349 consensus_queue.cc:237] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:00.712388 27652 raft_consensus.cc:1261] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:00.712869 30156 raft_consensus.cc:1261] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:00.712960 30421 raft_consensus.cc:1261] T 8bb679bf525a4a2ebafeb73740230569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:00.713331 31349 consensus_queue.cc:1035] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:00.713619 31348 consensus_queue.cc:1035] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:00.715682 31348 consensus_queue.cc:1035] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:00.719717 31348 raft_consensus.cc:2932] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:00.722673 27652 raft_consensus.cc:2932] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:00.725497 30156 raft_consensus.cc:2932] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:00.731417 27116 catalog_manager.cc:5484] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:00.736341 30028 raft_consensus.cc:1203] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:00.752426 30421 raft_consensus.cc:2932] T 8bb679bf525a4a2ebafeb73740230569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:00.761178 30028 consensus_queue.cc:237] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:00.763448 27652 raft_consensus.cc:1261] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:00.764022 31349 consensus_queue.cc:1035] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:00.765035 30423 raft_consensus.cc:1261] T 8bb679bf525a4a2ebafeb73740230569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:00.765583 31349 consensus_queue.cc:1035] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:00.766664 31348 raft_consensus.cc:2932] T 8bb679bf525a4a2ebafeb73740230569 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:00.770517 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8bb679bf525a4a2ebafeb73740230569 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:00.770617 27652 raft_consensus.cc:2932] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:00.777067 30421 raft_consensus.cc:1203] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:00.778681 27109 catalog_manager.cc:5484] T 8bb679bf525a4a2ebafeb73740230569 P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:00.781536 30423 raft_consensus.cc:2932] T 8bb679bf525a4a2ebafeb73740230569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:00.786170 30139 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb679bf525a4a2ebafeb73740230569 with delete_type TABLET_DATA_TOMBSTONED (TS 84cda20738d2452da166e31fc6da57a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:00.786844 31650 tablet_replica.cc:330] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:00.787124 31650 raft_consensus.cc:2229] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:00.787628 31650 raft_consensus.cc:2258] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:00.789587 31650 ts_tablet_manager.cc:1875] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:00.792173 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:00.797741 30423 raft_consensus.cc:1261] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:00.798286 31348 consensus_queue.cc:1035] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:00.802181 30421 raft_consensus.cc:1203] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:00.804879 27110 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet c4f8db1d11ec4e59a8eaa8b4b80c1024 (table rebalancer_dynamic_table [id=a8189a946a094e6394d4cf6e22461d0d]). Expected version 1 got 0
I20240502 08:18:00.807444 31654 tablet.cc:1722] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:00.808794 31654 tablet.cc:1620] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:00.813505 31650 ts_tablet_manager.cc:1888] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:00.813684 31650 log.cc:1196] T 8bb679bf525a4a2ebafeb73740230569 P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/8bb679bf525a4a2ebafeb73740230569
I20240502 08:18:00.814525 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet 8bb679bf525a4a2ebafeb73740230569 (table rebalancer_extra_table_6 [id=e57c96062aca4609bc2d923cb7a30a29]) successfully deleted
I20240502 08:18:00.814961 30288 raft_consensus.cc:1261] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:00.815615 31349 consensus_queue.cc:1035] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:00.821216 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=a8189a946a094e6394d4cf6e22461d0d] alter complete (version 1)
I20240502 08:18:00.825673 31656 tablet.cc:1722] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:00.828198 31656 tablet.cc:1620] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:00.828612 27110 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet c4f8db1d11ec4e59a8eaa8b4b80c1024 (table rebalancer_dynamic_table [id=a8189a946a094e6394d4cf6e22461d0d]). Expected version 1 got 0
I20240502 08:18:00.836232 27110 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:00.838490 31661 tablet.cc:1722] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:00.839179 31661 tablet.cc:1620] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:00.842396 27109 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet c4f8db1d11ec4e59a8eaa8b4b80c1024 (table rebalancer_dynamic_table [id=a8189a946a094e6394d4cf6e22461d0d]). Expected version 1 got 0
I20240502 08:18:00.848343 30139 tablet_service.cc:1467] Processing CreateTablet for tablet b9d6207b6bb3436f996a069ff910a56e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=98426515f8594fe1951ef5a81f079c16]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:00.848735 30139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d6207b6bb3436f996a069ff910a56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.848678 30402 tablet_service.cc:1467] Processing CreateTablet for tablet b9d6207b6bb3436f996a069ff910a56e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=98426515f8594fe1951ef5a81f079c16]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:00.849092 30402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d6207b6bb3436f996a069ff910a56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.850675 30400 tablet_service.cc:1467] Processing CreateTablet for tablet e7064865693149b980ba54fb8e65bc48 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=98426515f8594fe1951ef5a81f079c16]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:00.851217 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7064865693149b980ba54fb8e65bc48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.853336 30401 tablet_service.cc:1467] Processing CreateTablet for tablet a40b618e4cb9426c902c3384dea3bb46 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=98426515f8594fe1951ef5a81f079c16]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:00.853685 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40b618e4cb9426c902c3384dea3bb46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.854751 30007 tablet_service.cc:1467] Processing CreateTablet for tablet a40b618e4cb9426c902c3384dea3bb46 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=98426515f8594fe1951ef5a81f079c16]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:00.855113 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40b618e4cb9426c902c3384dea3bb46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.860034 31589 tablet_bootstrap.cc:492] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:00.861352 31589 tablet_bootstrap.cc:654] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.861889 30138 tablet_service.cc:1467] Processing CreateTablet for tablet a40b618e4cb9426c902c3384dea3bb46 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=98426515f8594fe1951ef5a81f079c16]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:00.862329 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40b618e4cb9426c902c3384dea3bb46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.863034 30140 tablet_service.cc:1467] Processing CreateTablet for tablet e7064865693149b980ba54fb8e65bc48 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=98426515f8594fe1951ef5a81f079c16]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:00.863490 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7064865693149b980ba54fb8e65bc48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.866501 27370 tablet_service.cc:1467] Processing CreateTablet for tablet b9d6207b6bb3436f996a069ff910a56e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=98426515f8594fe1951ef5a81f079c16]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:00.866961 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d6207b6bb3436f996a069ff910a56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.867722 31668 tablet_bootstrap.cc:492] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:00.869378 31668 tablet_bootstrap.cc:654] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.870220 27238 tablet_service.cc:1467] Processing CreateTablet for tablet e7064865693149b980ba54fb8e65bc48 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=98426515f8594fe1951ef5a81f079c16]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:00.870676 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7064865693149b980ba54fb8e65bc48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.871519 31590 tablet_bootstrap.cc:492] T a40b618e4cb9426c902c3384dea3bb46 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:00.873594 31590 tablet_bootstrap.cc:654] T a40b618e4cb9426c902c3384dea3bb46 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.879745 31671 tablet_bootstrap.cc:492] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:00.881219 31590 tablet_bootstrap.cc:492] T a40b618e4cb9426c902c3384dea3bb46 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:00.881366 31590 ts_tablet_manager.cc:1367] T a40b618e4cb9426c902c3384dea3bb46 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240502 08:18:00.882253 31590 raft_consensus.cc:348] T a40b618e4cb9426c902c3384dea3bb46 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:00.882542 31590 raft_consensus.cc:374] T a40b618e4cb9426c902c3384dea3bb46 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.882617 31590 raft_consensus.cc:729] T a40b618e4cb9426c902c3384dea3bb46 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.883620 31671 tablet_bootstrap.cc:654] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.883824 31590 consensus_queue.cc:260] T a40b618e4cb9426c902c3384dea3bb46 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:00.884410 31590 ts_tablet_manager.cc:1398] T a40b618e4cb9426c902c3384dea3bb46 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:00.884860 31672 tablet_bootstrap.cc:492] T b9d6207b6bb3436f996a069ff910a56e P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:00.887476 31589 tablet_bootstrap.cc:492] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:00.887638 31668 tablet_bootstrap.cc:492] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:00.887663 31589 ts_tablet_manager.cc:1367] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.002s
I20240502 08:18:00.887805 31668 ts_tablet_manager.cc:1367] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.003s
I20240502 08:18:00.888480 31589 raft_consensus.cc:348] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:00.888682 31589 raft_consensus.cc:374] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.888746 31589 raft_consensus.cc:729] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.888690 31668 raft_consensus.cc:348] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:00.888931 31668 raft_consensus.cc:374] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.889045 31668 raft_consensus.cc:729] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.888965 31589 consensus_queue.cc:260] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:00.889340 31668 consensus_queue.cc:260] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:00.889480 31589 ts_tablet_manager.cc:1398] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 08:18:00.889801 31589 tablet_bootstrap.cc:492] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:00.890074 31668 ts_tablet_manager.cc:1398] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:18:00.890358 31668 tablet_bootstrap.cc:492] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:00.892169 31668 tablet_bootstrap.cc:654] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.893329 31672 tablet_bootstrap.cc:654] T b9d6207b6bb3436f996a069ff910a56e P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.899744 31589 tablet_bootstrap.cc:654] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.900043 31672 tablet_bootstrap.cc:492] T b9d6207b6bb3436f996a069ff910a56e P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:18:00.900240 31672 ts_tablet_manager.cc:1367] T b9d6207b6bb3436f996a069ff910a56e P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240502 08:18:00.902709 31672 raft_consensus.cc:348] T b9d6207b6bb3436f996a069ff910a56e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:00.902982 31672 raft_consensus.cc:374] T b9d6207b6bb3436f996a069ff910a56e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.903133 31672 raft_consensus.cc:729] T b9d6207b6bb3436f996a069ff910a56e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.903422 31672 consensus_queue.cc:260] T b9d6207b6bb3436f996a069ff910a56e P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:00.906661 31599 raft_consensus.cc:482] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:00.906790 31599 raft_consensus.cc:504] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:00.907578 31599 leader_election.cc:288] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:00.913470 31672 ts_tablet_manager.cc:1398] T b9d6207b6bb3436f996a069ff910a56e P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 08:18:00.914048 30158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7064865693149b980ba54fb8e65bc48" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
W20240502 08:18:00.914577 30358 leader_election.cc:341] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:00.916776 31668 tablet_bootstrap.cc:492] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:00.916996 31668 ts_tablet_manager.cc:1367] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.014s
I20240502 08:18:00.917858 31668 raft_consensus.cc:348] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:00.918046 31589 tablet_bootstrap.cc:492] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:00.918089 31668 raft_consensus.cc:374] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.918202 31668 raft_consensus.cc:729] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.918210 31589 ts_tablet_manager.cc:1367] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.001s
I20240502 08:18:00.918470 31668 consensus_queue.cc:260] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:00.919088 31668 ts_tablet_manager.cc:1398] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:00.919137 31589 raft_consensus.cc:348] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:00.919342 31589 raft_consensus.cc:374] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.919441 31589 raft_consensus.cc:729] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.919543 31668 tablet_bootstrap.cc:492] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:00.919673 31589 consensus_queue.cc:260] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:00.921300 31671 tablet_bootstrap.cc:492] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:00.921350 31589 ts_tablet_manager.cc:1398] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:18:00.921499 31668 tablet_bootstrap.cc:654] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.921681 31589 tablet_bootstrap.cc:492] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:00.923487 31589 tablet_bootstrap.cc:654] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.933930 31671 ts_tablet_manager.cc:1367] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.021s
I20240502 08:18:00.935197 31671 raft_consensus.cc:348] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:00.935559 31671 raft_consensus.cc:374] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.936832 27256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7064865693149b980ba54fb8e65bc48" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:00.937072 27109 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:00.938498 31589 tablet_bootstrap.cc:492] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:00.938691 31589 ts_tablet_manager.cc:1367] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240502 08:18:00.939548 31589 raft_consensus.cc:348] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:00.939611 30871 raft_consensus.cc:1050] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 to VOTER
I20240502 08:18:00.939767 31589 raft_consensus.cc:374] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.939850 31589 raft_consensus.cc:729] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.939834 31113 raft_consensus.cc:482] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:00.939985 31113 raft_consensus.cc:504] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:00.940714 30871 consensus_queue.cc:237] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:00.941001 31113 leader_election.cc:288] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:00.941941 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d6207b6bb3436f996a069ff910a56e" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:00.942265 30423 raft_consensus.cc:2454] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:00.943681 31589 consensus_queue.cc:260] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:00.944190 30093 leader_election.cc:302] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:00.944579 31113 raft_consensus.cc:2781] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:00.944615 31589 ts_tablet_manager.cc:1398] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:18:00.944700 31113 raft_consensus.cc:482] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:00.944806 31113 raft_consensus.cc:3037] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.946130 31668 tablet_bootstrap.cc:492] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:00.946272 31668 ts_tablet_manager.cc:1367] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.012s
I20240502 08:18:00.947023 31668 raft_consensus.cc:348] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:00.947228 31668 raft_consensus.cc:374] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.947311 31668 raft_consensus.cc:729] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.947517 31668 consensus_queue.cc:260] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:00.948879 31668 ts_tablet_manager.cc:1398] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:00.956876 27108 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet c4f8db1d11ec4e59a8eaa8b4b80c1024 (table rebalancer_dynamic_table [id=a8189a946a094e6394d4cf6e22461d0d]). Expected version 2 got 1
I20240502 08:18:00.959475 31113 raft_consensus.cc:504] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:00.961889 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d6207b6bb3436f996a069ff910a56e" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:00.962136 30421 raft_consensus.cc:3037] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.965340 30421 raft_consensus.cc:2454] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:00.966065 30093 leader_election.cc:302] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:00.966769 31682 raft_consensus.cc:2781] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:00.966903 31682 raft_consensus.cc:686] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:00.967329 31113 leader_election.cc:288] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:00.971029 31654 tablet.cc:1722] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:00.935701 31671 raft_consensus.cc:729] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.971570 31654 tablet.cc:1620] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:00.971695 31671 consensus_queue.cc:260] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:00.972049 31656 tablet.cc:1722] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:00.972162 27256 raft_consensus.cc:2454] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:00.972237 31661 tablet.cc:1722] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:00.972550 31656 tablet.cc:1620] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:00.972695 31661 tablet.cc:1620] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:00.972797 30357 leader_election.cc:302] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:00.973206 31599 raft_consensus.cc:2781] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:00.973310 31599 raft_consensus.cc:482] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:00.973413 31599 raft_consensus.cc:3037] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.976270 31599 raft_consensus.cc:504] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:00.977432 27256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7064865693149b980ba54fb8e65bc48" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:00.977644 27256 raft_consensus.cc:3037] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.977579 30158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7064865693149b980ba54fb8e65bc48" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:00.977810 30158 raft_consensus.cc:3037] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.981361 30158 raft_consensus.cc:2454] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:00.967211 31682 consensus_queue.cc:237] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:00.982969 30358 leader_election.cc:302] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:00.983281 31309 raft_consensus.cc:2781] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:00.987541 30156 raft_consensus.cc:1203] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:00.987800 27256 raft_consensus.cc:2454] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:00.991540 31599 leader_election.cc:288] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:00.996073 27109 catalog_manager.cc:5484] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:00.999684 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=a8189a946a094e6394d4cf6e22461d0d] alter complete (version 2)
I20240502 08:18:01.006234 27521 raft_consensus.cc:1261] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.015818 31689 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.019251 31309 raft_consensus.cc:686] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:18:01.020783 31309 consensus_queue.cc:237] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:01.025419 30288 raft_consensus.cc:1261] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.026770 27650 raft_consensus.cc:1261] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.027403 31540 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.027503 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d6207b6bb3436f996a069ff910a56e" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:18:01.027714 27390 raft_consensus.cc:3037] T b9d6207b6bb3436f996a069ff910a56e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:01.028045 31540 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.030457 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d6207b6bb3436f996a069ff910a56e" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:18:01.031841 27390 raft_consensus.cc:2454] T b9d6207b6bb3436f996a069ff910a56e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:01.051295 30288 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:01.051813 31671 ts_tablet_manager.cc:1398] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.118s user 0.028s sys 0.024s
I20240502 08:18:01.051642 31687 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:01.052294 27110 catalog_manager.cc:5484] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:01.057243 27650 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:01.058245 27521 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:01.059362 27110 catalog_manager.cc:5484] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:01.066804 27256 consensus_queue.cc:237] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:01.069394 27650 raft_consensus.cc:1261] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:01.069841 31689 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:01.072594 31688 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:01.072902 27652 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:01.075284 30289 raft_consensus.cc:1261] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:01.075953 31680 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:01.076660 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 15a333c65455473fbdc26d1ec137aebc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:01.077505 30289 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:01.084582 27109 catalog_manager.cc:5484] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:01.089674 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet 15a333c65455473fbdc26d1ec137aebc with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:01.090397 31698 tablet_replica.cc:330] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:01.090878 31698 raft_consensus.cc:2229] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:01.092098 31698 raft_consensus.cc:2258] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:01.092947 31698 ts_tablet_manager.cc:1875] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:01.099474 31698 ts_tablet_manager.cc:1888] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:01.099638 31698 log.cc:1196] T 15a333c65455473fbdc26d1ec137aebc P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/15a333c65455473fbdc26d1ec137aebc
I20240502 08:18:01.100746 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 15a333c65455473fbdc26d1ec137aebc (table rebalance_test_table_1 [id=33e20eb07c524d63b8e99e7957467d8b]) successfully deleted
I20240502 08:18:01.104295 31618 raft_consensus.cc:1050] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442: attempting to promote NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 to VOTER
I20240502 08:18:01.105275 31618 consensus_queue.cc:237] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:01.107661 27390 raft_consensus.cc:1261] T 84953bd8daba4885be1f5f9ad4149aba P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:01.109612 31359 raft_consensus.cc:1050] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 to VOTER
I20240502 08:18:01.110363 31359 consensus_queue.cc:237] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:01.113232 31553 consensus_queue.cc:1035] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:01.114732 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:01.117633 30028 raft_consensus.cc:1261] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:01.118563 30289 raft_consensus.cc:1261] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:01.120617 31365 consensus_queue.cc:1035] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:01.121135 31365 consensus_queue.cc:1035] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:01.126834 27650 raft_consensus.cc:1261] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.127640 31654 tablet.cc:1722] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:01.127625 31350 consensus_queue.cc:1035] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.128717 30028 raft_consensus.cc:1261] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.134522 27390 raft_consensus.cc:1261] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.135653 31360 consensus_queue.cc:1035] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.136377 31360 consensus_queue.cc:1035] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.139931 31656 tablet.cc:1722] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:01.144021 31661 tablet.cc:1722] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:01.151616 30158 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ec6c977cac054b758a43d846e821707a"
dest_uuid: "84cda20738d2452da166e31fc6da57a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48882
I20240502 08:18:01.151917 30158 raft_consensus.cc:595] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:18:01.163111 27108 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=a8189a946a094e6394d4cf6e22461d0d] alter complete (version 3)
I20240502 08:18:01.167815 30288 raft_consensus.cc:2932] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:01.176434 31350 raft_consensus.cc:1050] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 to VOTER
I20240502 08:18:01.176221 27110 catalog_manager.cc:5484] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:01.177486 31350 consensus_queue.cc:237] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:01.179258 31666 raft_consensus.cc:482] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:01.179443 31666 raft_consensus.cc:504] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:01.180292 31666 leader_election.cc:288] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:01.181278 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40b618e4cb9426c902c3384dea3bb46" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:01.181555 30423 raft_consensus.cc:2454] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:01.182276 30093 leader_election.cc:302] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:01.182688 31666 raft_consensus.cc:2781] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:01.182813 31666 raft_consensus.cc:482] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:01.182909 31666 raft_consensus.cc:3037] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:01.186604 31666 raft_consensus.cc:504] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:01.188036 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40b618e4cb9426c902c3384dea3bb46" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:01.188347 30028 raft_consensus.cc:3037] T a40b618e4cb9426c902c3384dea3bb46 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:01.188578 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40b618e4cb9426c902c3384dea3bb46" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:01.188838 30423 raft_consensus.cc:3037] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:01.191969 30423 raft_consensus.cc:2454] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:01.192513 30028 raft_consensus.cc:2454] T a40b618e4cb9426c902c3384dea3bb46 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:01.192488 30093 leader_election.cc:302] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:01.192811 31682 raft_consensus.cc:2781] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:01.192974 31682 raft_consensus.cc:686] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:01.193269 31682 consensus_queue.cc:237] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:01.196206 31666 leader_election.cc:288] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:01.196597 30026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40b618e4cb9426c902c3384dea3bb46" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:01.197337 30026 raft_consensus.cc:2362] T a40b618e4cb9426c902c3384dea3bb46 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1. Re-sending same reply.
I20240502 08:18:01.198329 27256 raft_consensus.cc:1261] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.199033 27389 raft_consensus.cc:1261] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.200136 31360 consensus_queue.cc:1035] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.200359 30028 raft_consensus.cc:1261] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.200819 31360 consensus_queue.cc:1035] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.201511 31360 consensus_queue.cc:1035] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.202548 31618 raft_consensus.cc:2932] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:01.204751 30028 raft_consensus.cc:2932] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:01.218413 27648 consensus_queue.cc:237] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:01.220469 30288 raft_consensus.cc:1261] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:01.221134 31618 consensus_queue.cc:1035] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:01.225768 31553 raft_consensus.cc:2932] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:01.230777 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84953bd8daba4885be1f5f9ad4149aba with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:01.232309 30288 raft_consensus.cc:2932] T 84953bd8daba4885be1f5f9ad4149aba P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:01.233378 27110 catalog_manager.cc:5484] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:01.237656 30007 tablet_service.cc:1514] Processing DeleteTablet for tablet 84953bd8daba4885be1f5f9ad4149aba with delete_type TABLET_DATA_TOMBSTONED (TS 1f0bf4bc3c5146539146b825b048bff3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:01.240825 31722 tablet_replica.cc:330] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:01.241012 31722 raft_consensus.cc:2229] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:01.241276 31722 raft_consensus.cc:2258] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:01.242162 27108 catalog_manager.cc:5484] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:01.244261 31722 ts_tablet_manager.cc:1875] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 08:18:01.260646 27326 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.3:44469, user_credentials={real_user=slave}} blocked reactor thread for 99633.5us
I20240502 08:18:01.262349 27387 raft_consensus.cc:2932] T 84953bd8daba4885be1f5f9ad4149aba P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:01.264484 31722 ts_tablet_manager.cc:1888] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:01.264626 31722 log.cc:1196] T 84953bd8daba4885be1f5f9ad4149aba P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/84953bd8daba4885be1f5f9ad4149aba
I20240502 08:18:01.265338 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet 84953bd8daba4885be1f5f9ad4149aba (table rebalancer_extra_table_10 [id=9df4632350e6444089d25e0f410be77a]) successfully deleted
I20240502 08:18:01.271442 27387 raft_consensus.cc:1261] T 84953bd8daba4885be1f5f9ad4149aba P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:01.273104 31618 consensus_queue.cc:1035] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:01.273406 27109 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:01.273666 27109 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:01.275485 27387 raft_consensus.cc:2932] T 84953bd8daba4885be1f5f9ad4149aba P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:01.277595 27109 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet c4f8db1d11ec4e59a8eaa8b4b80c1024
I20240502 08:18:01.278661 30007 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f8db1d11ec4e59a8eaa8b4b80c1024 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:01 UTC) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:01.278959 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f8db1d11ec4e59a8eaa8b4b80c1024 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:01 UTC) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:01.279183 30400 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f8db1d11ec4e59a8eaa8b4b80c1024 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:01 UTC) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:01.279328 31722 tablet_replica.cc:330] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:01.279520 31722 raft_consensus.cc:2229] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:01.279920 31722 raft_consensus.cc:2258] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:01.280967 31722 ts_tablet_manager.cc:1875] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:01.280550 31718 raft_consensus.cc:2932] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:01.283104 27650 raft_consensus.cc:2932] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:01.284482 31682 raft_consensus.cc:979] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6: : Instructing follower bf8ac71b15f740a0a928911360773442 to start an election
I20240502 08:18:01.284616 31682 raft_consensus.cc:1067] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Signalling peer bf8ac71b15f740a0a928911360773442 to start an election
I20240502 08:18:01.286026 27648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ec6c977cac054b758a43d846e821707a"
dest_uuid: "bf8ac71b15f740a0a928911360773442"
from {username='slave'} at 127.20.207.6:47641
I20240502 08:18:01.286291 27648 raft_consensus.cc:482] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:01.286437 27648 raft_consensus.cc:3037] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:01.286806 31689 raft_consensus.cc:1050] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 to VOTER
I20240502 08:18:01.287876 31689 consensus_queue.cc:237] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:01.306107 27387 raft_consensus.cc:2932] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:01.306558 31728 raft_consensus.cc:2932] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
W20240502 08:18:01.314939 27453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.5:37815, user_credentials={real_user=slave}} blocked reactor thread for 114203us
I20240502 08:18:01.325075 31365 raft_consensus.cc:1050] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442: attempting to promote NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad to VOTER
I20240502 08:18:01.327268 31365 consensus_queue.cc:237] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:01.330691 27258 raft_consensus.cc:1261] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.330900 27389 raft_consensus.cc:2932] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:01.331494 31553 consensus_queue.cc:1035] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.331923 27519 raft_consensus.cc:1261] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.332695 30423 raft_consensus.cc:1261] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.333050 31618 consensus_queue.cc:1035] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.330485 27256 raft_consensus.cc:2932] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:01.335892 31618 consensus_queue.cc:1035] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.336715 27108 catalog_manager.cc:5484] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:01.343640 31726 tablet_replica.cc:330] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:01.321045 27648 raft_consensus.cc:504] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:01.352566 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6c977cac054b758a43d846e821707a" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:01.352752 30288 raft_consensus.cc:3037] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:01.355885 30288 raft_consensus.cc:2454] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 2.
I20240502 08:18:01.364024 27108 catalog_manager.cc:5484] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:01.366429 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6c977cac054b758a43d846e821707a" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:01.366614 27648 leader_election.cc:288] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:01.366675 27257 raft_consensus.cc:3037] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:01.369822 27257 raft_consensus.cc:2454] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 2.
I20240502 08:18:01.370872 27586 leader_election.cc:302] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 49c5c8f70e494edd9fcda40cc0af6cc4, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:01.371235 31618 raft_consensus.cc:2781] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:01.371398 31618 raft_consensus.cc:686] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [term 2 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:18:01.371678 31618 consensus_queue.cc:237] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:01.371807 30028 raft_consensus.cc:2932] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:01.381949 27109 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:01.343889 31726 raft_consensus.cc:2229] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:01.387923 31726 raft_consensus.cc:2258] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:01.389331 31726 ts_tablet_manager.cc:1875] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:01.390820 31722 ts_tablet_manager.cc:1888] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:01.390997 31722 log.cc:1196] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/c4f8db1d11ec4e59a8eaa8b4b80c1024
I20240502 08:18:01.391559 31722 ts_tablet_manager.cc:1909] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 1f0bf4bc3c5146539146b825b048bff3: Deleting consensus metadata
I20240502 08:18:01.392709 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet c4f8db1d11ec4e59a8eaa8b4b80c1024 (table rebalancer_dynamic_table_new_name [id=a8189a946a094e6394d4cf6e22461d0d]) successfully deleted
I20240502 08:18:01.401840 30158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6c977cac054b758a43d846e821707a" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:01.402091 30158 raft_consensus.cc:3032] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:01.402179 30158 raft_consensus.cc:729] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:01.402506 30158 consensus_queue.cc:260] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:01.402995 30158 raft_consensus.cc:3037] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:01.403159 30028 raft_consensus.cc:2932] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:01.403865 31726 ts_tablet_manager.cc:1888] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:01.404008 31726 log.cc:1196] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/c4f8db1d11ec4e59a8eaa8b4b80c1024
I20240502 08:18:01.404500 31726 ts_tablet_manager.cc:1909] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting consensus metadata
I20240502 08:18:01.405498 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet c4f8db1d11ec4e59a8eaa8b4b80c1024 (table rebalancer_dynamic_table_new_name [id=a8189a946a094e6394d4cf6e22461d0d]) successfully deleted
I20240502 08:18:01.405714 30158 raft_consensus.cc:2454] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 2.
I20240502 08:18:01.409006 31113 consensus_queue.cc:1035] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:01.413689 30140 tablet_service.cc:1467] Processing CreateTablet for tablet fdc6417f48234fb3947f11e74afcb845 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0d17e83423be4fbb8664730a0a4dc1dc]), partition=
I20240502 08:18:01.414091 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc6417f48234fb3947f11e74afcb845. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:01.415987 27238 tablet_service.cc:1467] Processing CreateTablet for tablet fdc6417f48234fb3947f11e74afcb845 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0d17e83423be4fbb8664730a0a4dc1dc]), partition=
I20240502 08:18:01.416324 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc6417f48234fb3947f11e74afcb845. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:01.417743 27519 consensus_queue.cc:237] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:01.418421 30268 tablet_service.cc:1467] Processing CreateTablet for tablet fdc6417f48234fb3947f11e74afcb845 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0d17e83423be4fbb8664730a0a4dc1dc]), partition=
I20240502 08:18:01.418825 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc6417f48234fb3947f11e74afcb845. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:01.423471 31671 tablet_bootstrap.cc:492] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:01.425117 31671 tablet_bootstrap.cc:654] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:01.427112 31553 raft_consensus.cc:1050] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442: attempting to promote NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad to VOTER
I20240502 08:18:01.430804 30028 raft_consensus.cc:1261] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:01.431357 31682 consensus_queue.cc:1035] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:01.427922 31553 consensus_queue.cc:237] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:01.434893 31359 consensus_queue.cc:1035] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:01.436121 31359 raft_consensus.cc:1050] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 84cda20738d2452da166e31fc6da57a6 to VOTER
I20240502 08:18:01.436892 31359 consensus_queue.cc:237] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:01.440507 31719 raft_consensus.cc:2932] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:01.442718 30026 raft_consensus.cc:2932] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:01.443637 27648 raft_consensus.cc:1261] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:01.444918 31668 tablet_bootstrap.cc:492] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:01.444954 31718 consensus_queue.cc:1035] T b82841ff10f24cf7a0cf490be3defe69 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:01.446558 31668 tablet_bootstrap.cc:654] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:01.450605 27109 catalog_manager.cc:5484] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 5 to 6, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:01.454746 27370 tablet_service.cc:1514] Processing DeleteTablet for tablet b82841ff10f24cf7a0cf490be3defe69 with delete_type TABLET_DATA_TOMBSTONED (TS 83e171a5e16c4008923f1d43fc5ce7b1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40868
I20240502 08:18:01.455235 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b82841ff10f24cf7a0cf490be3defe69 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:01.461745 31746 tablet_replica.cc:330] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:18:01.463322 31746 raft_consensus.cc:2229] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:01.464278 27256 raft_consensus.cc:1261] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.464882 31359 consensus_queue.cc:1035] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.465281 31746 raft_consensus.cc:2258] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:01.465317 27648 raft_consensus.cc:2932] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:01.466063 30158 raft_consensus.cc:1261] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.465691 27650 raft_consensus.cc:1261] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.467172 31728 consensus_queue.cc:1035] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.467856 31728 consensus_queue.cc:1035] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.472075 31746 ts_tablet_manager.cc:1875] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:01.487560 31746 ts_tablet_manager.cc:1888] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:01.487802 31746 log.cc:1196] T b82841ff10f24cf7a0cf490be3defe69 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/b82841ff10f24cf7a0cf490be3defe69
I20240502 08:18:01.488692 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet b82841ff10f24cf7a0cf490be3defe69 (table rebalance_test_table_7 [id=83c398ec83c74ebf84113a0334649e63]) successfully deleted
I20240502 08:18:01.492583 30288 raft_consensus.cc:1261] T 3047a6db2c9d438eb5720d848b9ea5bd P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.493072 31680 consensus_queue.cc:1035] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.495332 27519 raft_consensus.cc:1261] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.495977 31680 consensus_queue.cc:1035] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.496873 31365 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:01.497444 27108 catalog_manager.cc:5484] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 1 to 2, leader changed from 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 2 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:01.504585 27648 raft_consensus.cc:1261] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.505508 30423 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:01.506587 31687 consensus_queue.cc:1035] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.508293 31671 tablet_bootstrap.cc:492] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:01.508496 31671 ts_tablet_manager.cc:1367] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.013s
I20240502 08:18:01.508858 27521 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:01.509471 31671 raft_consensus.cc:348] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:01.509768 31671 raft_consensus.cc:374] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:01.509826 31671 raft_consensus.cc:729] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:01.510064 31671 consensus_queue.cc:260] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:01.511458 31671 ts_tablet_manager.cc:1398] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:18:01.512459 31744 tablet_bootstrap.cc:492] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:01.514107 31744 tablet_bootstrap.cc:654] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:01.517138 31359 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:01.520483 27109 catalog_manager.cc:5484] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:01.522868 31309 consensus_queue.cc:1035] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:18:01.523514 31668 tablet_bootstrap.cc:492] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:01.523682 31668 ts_tablet_manager.cc:1367] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.079s user 0.006s sys 0.006s
I20240502 08:18:01.523782 27256 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:01.524407 31668 raft_consensus.cc:348] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:01.524600 31668 raft_consensus.cc:374] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:01.524682 31668 raft_consensus.cc:729] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:01.524893 31668 consensus_queue.cc:260] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:01.524868 27255 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:01.525382 27648 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:01.528824 30158 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:01.531409 31725 tablet_replica.cc:330] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:01.531607 31725 raft_consensus.cc:2229] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:01.531811 31725 raft_consensus.cc:2258] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:01.536584 30422 raft_consensus.cc:1261] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.538475 31553 consensus_queue.cc:1035] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.540863 31668 ts_tablet_manager.cc:1398] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.017s user 0.000s sys 0.003s
I20240502 08:18:01.547323 27109 catalog_manager.cc:5484] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:01.547514 27648 consensus_queue.cc:237] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:01.548841 27521 raft_consensus.cc:1261] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.555358 30422 raft_consensus.cc:1261] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:01.553217 27520 consensus_queue.cc:237] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:01.554986 27518 raft_consensus.cc:1261] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:01.556178 31553 consensus_queue.cc:1035] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:01.558018 30158 raft_consensus.cc:1261] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:01.558087 27255 raft_consensus.cc:1261] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:01.558508 31728 consensus_queue.cc:1035] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:01.559082 31359 consensus_queue.cc:1035] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:01.559157 31553 consensus_queue.cc:1035] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:01.559367 31753 consensus_queue.cc:1035] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.559950 31751 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:01.561975 31359 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:01.564595 27387 raft_consensus.cc:1261] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:01.565342 31553 consensus_queue.cc:1035] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.566023 30156 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:01.566314 30422 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:01.569798 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb5345b931504cfdbb272189b719efa4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:01.570147 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a31709aa6b2b4ff784114fb48cbc8f2a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:01.573035 27520 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:01.574337 27108 catalog_manager.cc:5484] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:01.578974 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet a31709aa6b2b4ff784114fb48cbc8f2a with delete_type TABLET_DATA_TOMBSTONED (TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39166
I20240502 08:18:01.582934 27116 catalog_manager.cc:5484] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:01.591599 31759 tablet_replica.cc:330] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:18:01.592375 31759 raft_consensus.cc:2229] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:01.592959 31759 raft_consensus.cc:2258] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:01.595181 31759 ts_tablet_manager.cc:1875] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:01.596675 27255 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:01.598109 31365 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:01.601261 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet eb5345b931504cfdbb272189b719efa4 with delete_type TABLET_DATA_TOMBSTONED (TS bf8ac71b15f740a0a928911360773442 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44788
I20240502 08:18:01.602926 30422 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:01.603977 27518 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:01.609061 27109 catalog_manager.cc:5484] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:01.614317 31759 ts_tablet_manager.cc:1888] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:01.617782 31759 log.cc:1196] T a31709aa6b2b4ff784114fb48cbc8f2a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/a31709aa6b2b4ff784114fb48cbc8f2a
I20240502 08:18:01.618964 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet a31709aa6b2b4ff784114fb48cbc8f2a (table rebalance_test_table_17 [id=c179f8e244074d9ead49cbf91748c01e]) successfully deleted
I20240502 08:18:01.618804 27387 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:01.621009 31763 tablet_replica.cc:330] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:18:01.621392 31763 raft_consensus.cc:2229] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:01.622129 31763 raft_consensus.cc:2258] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:01.624038 27648 consensus_queue.cc:237] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:01.625591 30422 raft_consensus.cc:1261] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:01.626155 31618 consensus_queue.cc:1035] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:01.629225 31753 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:01.629940 30422 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:01.632997 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 33aee3df5eb242ef9cc056d25f6d6a82 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:01.638250 27108 catalog_manager.cc:5484] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:01.639993 31113 consensus_queue.cc:1035] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:01.643090 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet 33aee3df5eb242ef9cc056d25f6d6a82 with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:01.644372 31763 ts_tablet_manager.cc:1875] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:01.648849 31309 consensus_queue.cc:1035] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:01.657754 31763 ts_tablet_manager.cc:1888] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:01.660646 31763 log.cc:1196] T eb5345b931504cfdbb272189b719efa4 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/eb5345b931504cfdbb272189b719efa4
I20240502 08:18:01.661777 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet eb5345b931504cfdbb272189b719efa4 (table rebalance_test_table_15 [id=f2de4023ae664751b94f83ff4434cd92]) successfully deleted
I20240502 08:18:01.687822 31725 ts_tablet_manager.cc:1875] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:01.691183 31744 tablet_bootstrap.cc:492] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:01.691421 31744 ts_tablet_manager.cc:1367] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.179s user 0.003s sys 0.000s
I20240502 08:18:01.692485 31744 raft_consensus.cc:348] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:01.692826 31744 raft_consensus.cc:374] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:01.692973 31744 raft_consensus.cc:729] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:01.693256 31744 consensus_queue.cc:260] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:01.699486 31113 consensus_queue.cc:1035] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:01.714866 27520 consensus_queue.cc:237] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:01.717201 30026 raft_consensus.cc:1261] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:01.717494 27390 raft_consensus.cc:1261] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:01.718070 31359 consensus_queue.cc:1035] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:01.718411 31719 consensus_queue.cc:1035] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:01.721982 31728 raft_consensus.cc:2932] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:01.723681 27390 raft_consensus.cc:2932] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:01.725865 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f2b08a6b38f945b1b99f1d8ad1fd0ae0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:01.726498 30026 raft_consensus.cc:2932] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:01.735742 27389 raft_consensus.cc:1261] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:01.736598 31553 consensus_queue.cc:1035] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:01.740263 27389 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:01.744992 31540 raft_consensus.cc:482] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:01.745179 31540 raft_consensus.cc:504] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:01.745764 31540 leader_election.cc:288] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:01.747025 30158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc6417f48234fb3947f11e74afcb845" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:01.747305 30158 raft_consensus.cc:2454] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:18:01.747848 27194 leader_election.cc:302] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:01.748178 31540 raft_consensus.cc:2781] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:01.748298 31540 raft_consensus.cc:482] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:01.748404 31540 raft_consensus.cc:3037] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:01.751529 31540 raft_consensus.cc:504] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:01.752710 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc6417f48234fb3947f11e74afcb845" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:01.752930 30290 raft_consensus.cc:3037] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:01.752780 27108 catalog_manager.cc:5484] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:01.755584 30290 raft_consensus.cc:2454] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:18:01.757222 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet f2b08a6b38f945b1b99f1d8ad1fd0ae0 with delete_type TABLET_DATA_TOMBSTONED (TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39166
I20240502 08:18:01.757191 30158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc6417f48234fb3947f11e74afcb845" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:01.757400 30158 raft_consensus.cc:3037] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:01.757572 31759 tablet_replica.cc:330] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:18:01.757819 31759 raft_consensus.cc:2229] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:01.758081 31759 raft_consensus.cc:2258] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:01.759382 31759 ts_tablet_manager.cc:1875] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:01.760162 30158 raft_consensus.cc:2454] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:18:01.760874 27194 leader_election.cc:302] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:01.761137 31540 leader_election.cc:288] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:01.761374 31680 raft_consensus.cc:2781] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:01.761533 31680 raft_consensus.cc:686] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:18:01.761860 31680 consensus_queue.cc:237] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:01.763176 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc6417f48234fb3947f11e74afcb845" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:01.763370 30288 raft_consensus.cc:2362] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1. Re-sending same reply.
I20240502 08:18:01.777863 31759 ts_tablet_manager.cc:1888] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:01.779421 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet a31709aa6b2b4ff784114fb48cbc8f2a with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 6)) from {username='slave'} at 127.0.0.1:39166
I20240502 08:18:01.782346 31759 log.cc:1196] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/f2b08a6b38f945b1b99f1d8ad1fd0ae0
I20240502 08:18:01.784440 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet f2b08a6b38f945b1b99f1d8ad1fd0ae0 (table rebalance_test_table_1 [id=33e20eb07c524d63b8e99e7957467d8b]) successfully deleted
I20240502 08:18:01.784834 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet a31709aa6b2b4ff784114fb48cbc8f2a (table rebalance_test_table_17 [id=c179f8e244074d9ead49cbf91748c01e]) successfully deleted
I20240502 08:18:01.789055 27109 catalog_manager.cc:5484] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:01.795046 31744 ts_tablet_manager.cc:1398] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.103s user 0.002s sys 0.000s
I20240502 08:18:01.798555 31687 raft_consensus.cc:2932] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:01.800432 31765 tablet_replica.cc:330] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:01.800660 31765 raft_consensus.cc:2229] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:01.801018 31765 raft_consensus.cc:2258] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:01.802819 27650 raft_consensus.cc:2932] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:01.805433 27116 catalog_manager.cc:5484] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:01.807924 27519 raft_consensus.cc:2932] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:01.809037 30288 raft_consensus.cc:1261] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:18:01.809610 31753 consensus_queue.cc:1035] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:01.822355 31765 ts_tablet_manager.cc:1875] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:01.827908 31725 ts_tablet_manager.cc:1888] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:01.830031 31725 log.cc:1196] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/c4f8db1d11ec4e59a8eaa8b4b80c1024
I20240502 08:18:01.830650 31725 ts_tablet_manager.cc:1909] T c4f8db1d11ec4e59a8eaa8b4b80c1024 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting consensus metadata
I20240502 08:18:01.832000 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet c4f8db1d11ec4e59a8eaa8b4b80c1024 (table rebalancer_dynamic_table_new_name [id=a8189a946a094e6394d4cf6e22461d0d]) successfully deleted
I20240502 08:18:01.833174 27108 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:01.840961 30288 raft_consensus.cc:2932] T 3047a6db2c9d438eb5720d848b9ea5bd P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:01.846577 30158 raft_consensus.cc:1261] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:18:01.847335 31553 consensus_queue.cc:1035] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:01.852505 30400 tablet_service.cc:1467] Processing CreateTablet for tablet 2abc11a238a542e4982452b5ce341101 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2c7cf528be404fbca449af1fcf7d0e23]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:01.852768 30007 tablet_service.cc:1467] Processing CreateTablet for tablet 2abc11a238a542e4982452b5ce341101 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2c7cf528be404fbca449af1fcf7d0e23]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:01.852943 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2abc11a238a542e4982452b5ce341101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:01.853178 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2abc11a238a542e4982452b5ce341101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:01.856851 27500 tablet_service.cc:1467] Processing CreateTablet for tablet 2abc11a238a542e4982452b5ce341101 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2c7cf528be404fbca449af1fcf7d0e23]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:01.857333 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2abc11a238a542e4982452b5ce341101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:01.858328 30401 tablet_service.cc:1467] Processing CreateTablet for tablet eca2618572fe4900bd420bb59c22381f (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2c7cf528be404fbca449af1fcf7d0e23]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:01.858371 30140 tablet_service.cc:1467] Processing CreateTablet for tablet eca2618572fe4900bd420bb59c22381f (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2c7cf528be404fbca449af1fcf7d0e23]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:01.858815 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca2618572fe4900bd420bb59c22381f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:01.858903 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca2618572fe4900bd420bb59c22381f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:01.862423 30138 tablet_service.cc:1467] Processing CreateTablet for tablet a9e5e67e3fe54491841b33d0d4bd047b (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2c7cf528be404fbca449af1fcf7d0e23]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:01.862819 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e5e67e3fe54491841b33d0d4bd047b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:01.863358 27238 tablet_service.cc:1467] Processing CreateTablet for tablet a9e5e67e3fe54491841b33d0d4bd047b (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2c7cf528be404fbca449af1fcf7d0e23]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:01.863871 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e5e67e3fe54491841b33d0d4bd047b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:01.867352 27258 consensus_queue.cc:237] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:01.867511 30008 tablet_service.cc:1467] Processing CreateTablet for tablet eca2618572fe4900bd420bb59c22381f (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2c7cf528be404fbca449af1fcf7d0e23]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:01.868009 30008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca2618572fe4900bd420bb59c22381f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:01.870195 27519 raft_consensus.cc:1261] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:01.870397 30006 tablet_service.cc:1467] Processing CreateTablet for tablet a9e5e67e3fe54491841b33d0d4bd047b (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=2c7cf528be404fbca449af1fcf7d0e23]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:01.870770 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e5e67e3fe54491841b33d0d4bd047b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:01.870980 31680 consensus_queue.cc:1035] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:01.873533 31687 raft_consensus.cc:2932] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:01.874508 27520 raft_consensus.cc:2932] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:01.881421 27116 catalog_manager.cc:5484] T 3047a6db2c9d438eb5720d848b9ea5bd P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:01.881949 31671 tablet_bootstrap.cc:492] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:01.882525 30289 raft_consensus.cc:1261] T 3047a6db2c9d438eb5720d848b9ea5bd P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:01.883049 31680 consensus_queue.cc:1035] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:01.883594 31671 tablet_bootstrap.cc:654] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:01.884562 30288 raft_consensus.cc:2932] T 3047a6db2c9d438eb5720d848b9ea5bd P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:01.889968 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet 3047a6db2c9d438eb5720d848b9ea5bd with delete_type TABLET_DATA_TOMBSTONED (TS bf8ac71b15f740a0a928911360773442 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44788
I20240502 08:18:01.890322 31763 tablet_replica.cc:330] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:18:01.890499 31763 raft_consensus.cc:2229] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:01.890808 31763 raft_consensus.cc:2258] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:01.891963 31763 ts_tablet_manager.cc:1875] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:01.896852 31765 ts_tablet_manager.cc:1888] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:01.897257 31765 log.cc:1196] T 33aee3df5eb242ef9cc056d25f6d6a82 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/33aee3df5eb242ef9cc056d25f6d6a82
I20240502 08:18:01.898479 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 33aee3df5eb242ef9cc056d25f6d6a82 (table rebalance_test_table_9 [id=a65036830fa64a22b9f4efd6d5bd4f9f]) successfully deleted
I20240502 08:18:01.903312 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3047a6db2c9d438eb5720d848b9ea5bd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:01.912729 31671 tablet_bootstrap.cc:492] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:01.913007 31671 ts_tablet_manager.cc:1367] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.002s
I20240502 08:18:01.914366 31671 raft_consensus.cc:348] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:01.914705 31671 raft_consensus.cc:374] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:01.914861 31671 raft_consensus.cc:729] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:01.915149 31671 consensus_queue.cc:260] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:01.916146 31671 ts_tablet_manager.cc:1398] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:18:01.916900 31780 tablet_bootstrap.cc:492] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:01.919766 31778 tablet_bootstrap.cc:492] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:01.923307 31780 tablet_bootstrap.cc:654] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:01.926272 31763 ts_tablet_manager.cc:1888] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:01.928231 31668 tablet_bootstrap.cc:492] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:01.930753 31668 tablet_bootstrap.cc:654] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:01.934741 31778 tablet_bootstrap.cc:654] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:01.937520 27258 raft_consensus.cc:1261] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:18:01.938115 31618 consensus_queue.cc:1035] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:01.941282 31763 log.cc:1196] T 3047a6db2c9d438eb5720d848b9ea5bd P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/3047a6db2c9d438eb5720d848b9ea5bd
I20240502 08:18:01.941691 31778 tablet_bootstrap.cc:492] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:01.941855 31778 ts_tablet_manager.cc:1367] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.005s
I20240502 08:18:01.942703 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 3047a6db2c9d438eb5720d848b9ea5bd (table rebalance_test_table_17 [id=c179f8e244074d9ead49cbf91748c01e]) successfully deleted
I20240502 08:18:01.948822 31778 raft_consensus.cc:348] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:01.949465 31778 raft_consensus.cc:374] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:01.949585 31778 raft_consensus.cc:729] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:01.949873 31778 consensus_queue.cc:260] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:01.958922 31778 ts_tablet_manager.cc:1398] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20240502 08:18:01.959429 31780 tablet_bootstrap.cc:492] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:01.959589 31780 ts_tablet_manager.cc:1367] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.005s
I20240502 08:18:01.960337 31780 raft_consensus.cc:348] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:01.960551 31780 raft_consensus.cc:374] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:01.960666 31780 raft_consensus.cc:729] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:01.960889 31780 consensus_queue.cc:260] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:01.961526 31780 ts_tablet_manager.cc:1398] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:01.963977 31781 tablet_bootstrap.cc:492] T eca2618572fe4900bd420bb59c22381f P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:01.965138 27108 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:01.966838 31781 tablet_bootstrap.cc:654] T eca2618572fe4900bd420bb59c22381f P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:01.971659 31780 tablet_bootstrap.cc:492] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:01.972506 31668 tablet_bootstrap.cc:492] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:01.972638 30289 raft_consensus.cc:1261] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:01.972712 31668 ts_tablet_manager.cc:1367] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.002s
I20240502 08:18:01.973371 31780 tablet_bootstrap.cc:654] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:01.973455 31689 consensus_queue.cc:1035] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:01.973536 31668 raft_consensus.cc:348] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:01.973729 31668 raft_consensus.cc:374] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:01.973836 31668 raft_consensus.cc:729] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:01.975404 27650 consensus_queue.cc:237] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:01.975884 31668 consensus_queue.cc:260] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:01.977696 27258 raft_consensus.cc:1261] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:18:01.979437 31668 ts_tablet_manager.cc:1398] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 08:18:01.979774 31668 tablet_bootstrap.cc:492] T a9e5e67e3fe54491841b33d0d4bd047b P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:01.981837 31753 consensus_queue.cc:1035] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.984030 31668 tablet_bootstrap.cc:654] T a9e5e67e3fe54491841b33d0d4bd047b P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:01.985095 30288 raft_consensus.cc:1261] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:18:01.985733 31753 consensus_queue.cc:1035] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:01.989634 31753 raft_consensus.cc:2932] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:01.994161 27258 raft_consensus.cc:2932] T ec6c977cac054b758a43d846e821707a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:01.995652 31780 tablet_bootstrap.cc:492] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:01.995843 31780 ts_tablet_manager.cc:1367] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.000s
I20240502 08:18:01.996953 31780 raft_consensus.cc:348] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:01.997176 31780 raft_consensus.cc:374] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:01.997256 31780 raft_consensus.cc:729] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:01.997766 30158 raft_consensus.cc:1261] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:01.999064 31540 consensus_queue.cc:1035] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:01.999912 27116 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet fdc6417f48234fb3947f11e74afcb845 (table rebalancer_dynamic_table [id=0d17e83423be4fbb8664730a0a4dc1dc]). Expected version 1 got 0
I20240502 08:18:02.003937 31668 tablet_bootstrap.cc:492] T a9e5e67e3fe54491841b33d0d4bd047b P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:02.004109 31668 ts_tablet_manager.cc:1367] T a9e5e67e3fe54491841b33d0d4bd047b P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20240502 08:18:02.005092 31668 raft_consensus.cc:348] T a9e5e67e3fe54491841b33d0d4bd047b P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:02.005462 31668 raft_consensus.cc:374] T a9e5e67e3fe54491841b33d0d4bd047b P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:02.005620 31668 raft_consensus.cc:729] T a9e5e67e3fe54491841b33d0d4bd047b P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:02.005961 31668 consensus_queue.cc:260] T a9e5e67e3fe54491841b33d0d4bd047b P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:02.006683 31668 ts_tablet_manager.cc:1398] T a9e5e67e3fe54491841b33d0d4bd047b P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:02.006774 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ec6c977cac054b758a43d846e821707a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:02.007040 31780 consensus_queue.cc:260] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:02.008083 31780 ts_tablet_manager.cc:1398] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240502 08:18:02.008391 30289 raft_consensus.cc:2932] T ec6c977cac054b758a43d846e821707a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:02.008957 27109 catalog_manager.cc:5484] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 3 to 5, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New cstate: current_term: 2 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:02.012717 30140 tablet_service.cc:1514] Processing DeleteTablet for tablet ec6c977cac054b758a43d846e821707a with delete_type TABLET_DATA_TOMBSTONED (TS 84cda20738d2452da166e31fc6da57a6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:02.021775 31781 tablet_bootstrap.cc:492] T eca2618572fe4900bd420bb59c22381f P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:02.022255 31781 ts_tablet_manager.cc:1367] T eca2618572fe4900bd420bb59c22381f P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.005s
I20240502 08:18:02.024900 27110 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet fdc6417f48234fb3947f11e74afcb845 (table rebalancer_dynamic_table [id=0d17e83423be4fbb8664730a0a4dc1dc]). Expected version 1 got 0
I20240502 08:18:02.024834 31795 tablet.cc:1722] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:02.028144 27110 catalog_manager.cc:5576] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505) does not have the latest schema for tablet fdc6417f48234fb3947f11e74afcb845 (table rebalancer_dynamic_table [id=0d17e83423be4fbb8664730a0a4dc1dc]). Expected version 1 got 0
I20240502 08:18:02.029898 31781 raft_consensus.cc:348] T eca2618572fe4900bd420bb59c22381f P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:02.030165 31781 raft_consensus.cc:374] T eca2618572fe4900bd420bb59c22381f P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:02.030287 31781 raft_consensus.cc:729] T eca2618572fe4900bd420bb59c22381f P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:02.031013 31781 consensus_queue.cc:260] T eca2618572fe4900bd420bb59c22381f P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:02.033630 31796 tablet_replica.cc:330] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:02.034368 31781 ts_tablet_manager.cc:1398] T eca2618572fe4900bd420bb59c22381f P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 08:18:02.035226 31781 tablet_bootstrap.cc:492] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:02.035534 31802 tablet.cc:1722] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:02.036128 31802 tablet.cc:1620] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:18:02.036543 31796 raft_consensus.cc:2229] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:02.037043 31795 tablet.cc:1620] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: MemRowSet was empty: no flush needed.
I20240502 08:18:02.037712 31796 raft_consensus.cc:2258] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:02.040282 27110 catalog_manager.cc:5576] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899) does not have the latest schema for tablet fdc6417f48234fb3947f11e74afcb845 (table rebalancer_dynamic_table [id=0d17e83423be4fbb8664730a0a4dc1dc]). Expected version 1 got 0
I20240502 08:18:02.040879 31796 ts_tablet_manager.cc:1875] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:02.043797 31781 tablet_bootstrap.cc:654] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:02.045686 31798 tablet.cc:1722] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:02.047309 31798 tablet.cc:1620] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:02.048465 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=0d17e83423be4fbb8664730a0a4dc1dc] alter complete (version 1)
I20240502 08:18:02.050127 31796 ts_tablet_manager.cc:1888] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 08:18:02.050333 31796 log.cc:1196] T ec6c977cac054b758a43d846e821707a P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/ec6c977cac054b758a43d846e821707a
I20240502 08:18:02.051172 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet ec6c977cac054b758a43d846e821707a (table rebalancer_extra_table_8 [id=05e7c365fc854833a30f18097c117d10]) successfully deleted
I20240502 08:18:02.051460 31687 raft_consensus.cc:482] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:02.051604 31687 raft_consensus.cc:504] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:02.052263 31687 leader_election.cc:288] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:02.052834 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e5e67e3fe54491841b33d0d4bd047b" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:02.053335 30158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e5e67e3fe54491841b33d0d4bd047b" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:02.053552 30158 raft_consensus.cc:2454] T a9e5e67e3fe54491841b33d0d4bd047b P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:18:02.053947 27194 leader_election.cc:302] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:02.054232 31781 tablet_bootstrap.cc:492] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:02.054248 31687 raft_consensus.cc:2781] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:02.054345 31687 raft_consensus.cc:482] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:02.054406 31781 ts_tablet_manager.cc:1367] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20240502 08:18:02.054436 31687 raft_consensus.cc:3037] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:18:02.054899 27191 leader_election.cc:341] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:02.055249 31781 raft_consensus.cc:348] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:02.055471 31781 raft_consensus.cc:374] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:02.055562 31781 raft_consensus.cc:729] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:02.055804 31781 consensus_queue.cc:260] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:02.056437 31781 ts_tablet_manager.cc:1398] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:02.056820 31781 tablet_bootstrap.cc:492] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:02.056809 31687 raft_consensus.cc:504] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:02.057801 30158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e5e67e3fe54491841b33d0d4bd047b" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:02.057976 30158 raft_consensus.cc:3037] T a9e5e67e3fe54491841b33d0d4bd047b P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:02.057809 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e5e67e3fe54491841b33d0d4bd047b" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
W20240502 08:18:02.058806 27191 leader_election.cc:341] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:02.060652 31781 tablet_bootstrap.cc:654] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:02.065034 30158 raft_consensus.cc:2454] T a9e5e67e3fe54491841b33d0d4bd047b P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:18:02.065428 27194 leader_election.cc:302] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 84cda20738d2452da166e31fc6da57a6; no voters: 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:18:02.065953 31689 raft_consensus.cc:2781] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:02.066160 31689 raft_consensus.cc:686] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:18:02.066403 31689 consensus_queue.cc:237] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:02.067368 31687 leader_election.cc:288] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:02.071406 31781 tablet_bootstrap.cc:492] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:02.071545 31781 ts_tablet_manager.cc:1367] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.007s
I20240502 08:18:02.072201 31781 raft_consensus.cc:348] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:02.072352 31781 raft_consensus.cc:374] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:02.072445 31781 raft_consensus.cc:729] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:02.072654 31781 consensus_queue.cc:260] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:02.073146 31781 ts_tablet_manager.cc:1398] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 08:18:02.079257 27108 catalog_manager.cc:5484] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:02.120347 27258 consensus_queue.cc:237] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:02.122433 27650 raft_consensus.cc:1261] T 6978aa3955504c548778066ddd0910e5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:02.122531 27520 raft_consensus.cc:1261] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:02.123083 31687 consensus_queue.cc:1035] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:02.123394 31680 consensus_queue.cc:1035] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:18:02.126921 27191 consensus_peers.cc:485] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Couldn't send request to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6978aa3955504c548778066ddd0910e5. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:02.126711 31687 raft_consensus.cc:2932] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:02.131312 27650 raft_consensus.cc:2932] T 6978aa3955504c548778066ddd0910e5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:02.135286 27520 raft_consensus.cc:2932] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:02.144482 27110 catalog_manager.cc:5484] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:02.153517 31682 raft_consensus.cc:482] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:02.153712 31682 raft_consensus.cc:504] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:02.154526 31682 leader_election.cc:288] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:02.155833 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca2618572fe4900bd420bb59c22381f" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:02.156119 30028 raft_consensus.cc:2454] T eca2618572fe4900bd420bb59c22381f P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:02.156781 30093 leader_election.cc:302] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:02.156795 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca2618572fe4900bd420bb59c22381f" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:02.157119 30422 raft_consensus.cc:2454] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:02.158069 31682 raft_consensus.cc:2781] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:02.158203 31682 raft_consensus.cc:482] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:02.158324 31682 raft_consensus.cc:3037] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:02.164218 31682 raft_consensus.cc:504] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:02.165155 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca2618572fe4900bd420bb59c22381f" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:02.165220 31682 leader_election.cc:288] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:02.165336 30422 raft_consensus.cc:3037] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:02.165693 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca2618572fe4900bd420bb59c22381f" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:02.165890 30028 raft_consensus.cc:3037] T eca2618572fe4900bd420bb59c22381f P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:02.168006 30422 raft_consensus.cc:2454] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:02.168604 30028 raft_consensus.cc:2454] T eca2618572fe4900bd420bb59c22381f P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:02.168879 30093 leader_election.cc:302] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:02.169140 31682 raft_consensus.cc:2781] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:02.169467 31682 raft_consensus.cc:686] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:02.169739 31682 consensus_queue.cc:237] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:02.175151 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:02.178807 27520 consensus_queue.cc:237] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:02.182082 27387 raft_consensus.cc:1261] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:18:02.182242 27453 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09aa1b5d99114b3095be08c9547fbb54. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:02.182888 27108 catalog_manager.cc:5484] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:02.183723 31719 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:02.184628 31795 tablet.cc:1722] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:02.185079 31795 tablet.cc:1620] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: MemRowSet was empty: no flush needed.
I20240502 08:18:02.187754 31359 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:02.188328 31798 tablet.cc:1722] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:02.188766 31798 tablet.cc:1620] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:02.189589 27258 raft_consensus.cc:1261] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:02.190157 31705 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:02.191143 31802 tablet.cc:1722] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:02.191579 31802 tablet.cc:1620] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:18:02.193535 27387 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:02.194576 27108 catalog_manager.cc:5484] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:02.201867 31705 raft_consensus.cc:482] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:02.202091 31705 raft_consensus.cc:504] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:02.202765 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=0d17e83423be4fbb8664730a0a4dc1dc] alter complete (version 2)
I20240502 08:18:02.203065 31705 leader_election.cc:288] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:02.211195 27258 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:02.213198 27110 catalog_manager.cc:5576] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505) does not have the latest schema for tablet fdc6417f48234fb3947f11e74afcb845 (table rebalancer_dynamic_table [id=0d17e83423be4fbb8664730a0a4dc1dc]). Expected version 2 got 1
I20240502 08:18:02.228960 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abc11a238a542e4982452b5ce341101" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:02.229249 30028 raft_consensus.cc:2454] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:18:02.231003 27453 leader_election.cc:302] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:02.231698 31705 raft_consensus.cc:2781] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:02.231801 31705 raft_consensus.cc:482] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:02.231901 31705 raft_consensus.cc:3037] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:02.235673 31705 raft_consensus.cc:504] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:02.237457 31705 leader_election.cc:288] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:02.238058 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abc11a238a542e4982452b5ce341101" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:02.238267 30028 raft_consensus.cc:3037] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:02.241102 30028 raft_consensus.cc:2454] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:18:02.241663 27453 leader_election.cc:302] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:02.242048 31705 raft_consensus.cc:2781] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:02.242179 31705 raft_consensus.cc:686] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:18:02.242424 31705 consensus_queue.cc:237] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:02.246548 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abc11a238a542e4982452b5ce341101" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:02.246809 30422 raft_consensus.cc:2454] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:18:02.246548 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abc11a238a542e4982452b5ce341101" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:02.247202 30421 raft_consensus.cc:3037] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:02.249958 30421 raft_consensus.cc:2454] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:18:02.252707 27110 catalog_manager.cc:5484] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:02.271281 27387 consensus_queue.cc:237] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:02.274421 27520 raft_consensus.cc:1261] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:18:02.274662 27323 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Couldn't send request to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04a0a58eb52f41bf818e5102996d09e0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:02.275056 31479 consensus_queue.cc:1035] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:02.277518 31479 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:02.277961 27520 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:02.282078 27258 raft_consensus.cc:1261] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:02.282934 31845 consensus_queue.cc:1035] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:02.284294 27110 catalog_manager.cc:5484] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:02.290365 27258 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:02.306202 27108 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:02.310915 27258 consensus_queue.cc:237] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:02.313344 27650 raft_consensus.cc:1261] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:02.314215 27520 raft_consensus.cc:1261] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:02.314541 31689 consensus_queue.cc:1035] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:18:02.315135 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Couldn't send request to peer 84cda20738d2452da166e31fc6da57a6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af2f11471c064f48924191fc2607d31c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:02.315608 31689 consensus_queue.cc:1035] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:02.317466 31795 tablet.cc:1722] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:02.322942 31798 tablet.cc:1722] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:02.324313 31802 tablet.cc:1722] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:02.324070 31540 raft_consensus.cc:2932] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:02.337009 27520 raft_consensus.cc:2932] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:02.336552 27108 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=0d17e83423be4fbb8664730a0a4dc1dc] alter complete (version 3)
I20240502 08:18:02.341140 27650 raft_consensus.cc:2932] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:02.343436 27116 catalog_manager.cc:5484] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:02.350571 30421 consensus_queue.cc:237] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
W20240502 08:18:02.352874 30357 consensus_peers.cc:485] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): Couldn't send request to peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83993ff2073842988937597dd3999d0d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:02.354370 30156 raft_consensus.cc:1261] T 83993ff2073842988937597dd3999d0d P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:02.354904 31864 consensus_queue.cc:1035] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:02.355062 30028 raft_consensus.cc:1261] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:02.355604 31864 consensus_queue.cc:1035] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:02.357812 31309 raft_consensus.cc:2932] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:02.365561 30156 raft_consensus.cc:2932] T 83993ff2073842988937597dd3999d0d P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:02.366528 27108 catalog_manager.cc:5484] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:02.377854 30028 raft_consensus.cc:2932] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:02.392692 27520 consensus_queue.cc:237] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:02.395033 27650 raft_consensus.cc:1261] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:18:02.395131 27453 consensus_peers.cc:485] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6dce110666a4a038f5f46d48a3ec7af. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:02.395339 27387 raft_consensus.cc:1261] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:02.395748 31359 consensus_queue.cc:1035] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:02.396046 31705 consensus_queue.cc:1035] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:02.398698 31705 raft_consensus.cc:2932] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:02.399269 27387 raft_consensus.cc:2932] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:02.401474 27648 raft_consensus.cc:2932] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:02.405130 27108 catalog_manager.cc:5484] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:02.426180 27648 consensus_queue.cc:237] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
W20240502 08:18:02.428628 27584 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Couldn't send request to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cad036d297bd45d7b7c5a1f19356906f. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:02.429551 27387 raft_consensus.cc:1261] T cad036d297bd45d7b7c5a1f19356906f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:02.429891 27520 raft_consensus.cc:1261] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:02.430137 31365 consensus_queue.cc:1035] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:02.430419 31553 consensus_queue.cc:1035] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:02.432883 31365 raft_consensus.cc:2932] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:02.433604 27520 raft_consensus.cc:2932] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:02.433645 27387 raft_consensus.cc:2932] T cad036d297bd45d7b7c5a1f19356906f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:02.439069 27116 catalog_manager.cc:5484] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:02.444399 27116 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:02.444808 27116 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:02.448714 27116 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet fdc6417f48234fb3947f11e74afcb845
I20240502 08:18:02.449379 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet fdc6417f48234fb3947f11e74afcb845 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:02 UTC) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:02.449734 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet fdc6417f48234fb3947f11e74afcb845 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:02 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:18:02.449985 31886 tablet_replica.cc:330] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:02.450203 31886 raft_consensus.cc:2229] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:02.450542 31886 raft_consensus.cc:2258] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:02.451714 30140 tablet_service.cc:1514] Processing DeleteTablet for tablet fdc6417f48234fb3947f11e74afcb845 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:02 UTC) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:02.451920 31886 ts_tablet_manager.cc:1875] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:02.452014 31796 tablet_replica.cc:330] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:02.452217 31796 raft_consensus.cc:2229] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:02.452418 31796 raft_consensus.cc:2258] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:02.453533 31796 ts_tablet_manager.cc:1875] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:02.458717 31886 ts_tablet_manager.cc:1888] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:02.458832 31886 log.cc:1196] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/fdc6417f48234fb3947f11e74afcb845
I20240502 08:18:02.459204 31796 ts_tablet_manager.cc:1888] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:02.459312 31886 ts_tablet_manager.cc:1909] T fdc6417f48234fb3947f11e74afcb845 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting consensus metadata
I20240502 08:18:02.459333 31796 log.cc:1196] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/fdc6417f48234fb3947f11e74afcb845
I20240502 08:18:02.459736 31796 ts_tablet_manager.cc:1909] T fdc6417f48234fb3947f11e74afcb845 P 84cda20738d2452da166e31fc6da57a6: Deleting consensus metadata
I20240502 08:18:02.461169 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet fdc6417f48234fb3947f11e74afcb845 (table rebalancer_dynamic_table_new_name [id=0d17e83423be4fbb8664730a0a4dc1dc]) successfully deleted
I20240502 08:18:02.461210 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet fdc6417f48234fb3947f11e74afcb845 (table rebalancer_dynamic_table_new_name [id=0d17e83423be4fbb8664730a0a4dc1dc]) successfully deleted
I20240502 08:18:02.463775 30028 raft_consensus.cc:3037] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:02.466339 27258 consensus_queue.cc:237] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:02.466540 31687 consensus_queue.cc:1035] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:02.469273 27520 raft_consensus.cc:1261] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:02.469779 27648 raft_consensus.cc:1261] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:02.469820 31687 consensus_queue.cc:1035] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:02.469938 31887 tablet_replica.cc:330] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
W20240502 08:18:02.470256 27191 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Couldn't send request to peer 1f0bf4bc3c5146539146b825b048bff3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd887cbb35c74e54a43d3dc6a2dde817. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:02.470299 31887 raft_consensus.cc:2229] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:02.470715 31887 raft_consensus.cc:2258] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:02.471935 31887 ts_tablet_manager.cc:1875] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:02.472646 31687 consensus_queue.cc:1035] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:02.473613 31540 raft_consensus.cc:2932] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:02.474889 27520 raft_consensus.cc:2932] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:02.479916 27116 catalog_manager.cc:5484] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:02.483749 31887 ts_tablet_manager.cc:1888] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:02.483959 31887 log.cc:1196] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/fdc6417f48234fb3947f11e74afcb845
I20240502 08:18:02.484570 31887 ts_tablet_manager.cc:1909] T fdc6417f48234fb3947f11e74afcb845 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:18:02.485844 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet fdc6417f48234fb3947f11e74afcb845 (table rebalancer_dynamic_table_new_name [id=0d17e83423be4fbb8664730a0a4dc1dc]) successfully deleted
I20240502 08:18:02.491412 31680 consensus_queue.cc:1035] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:02.492878 27648 raft_consensus.cc:2932] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:02.512470 27258 consensus_queue.cc:237] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:02.515764 30158 raft_consensus.cc:1261] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:02.516021 30289 raft_consensus.cc:1261] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:02.516857 31680 consensus_queue.cc:1035] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:02.518426 31680 consensus_queue.cc:1035] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:02.521739 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e67c99e57a73443da13527ed33326a4e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:02.535292 31689 raft_consensus.cc:2932] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:02.535869 30158 raft_consensus.cc:2932] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:02.535938 30288 raft_consensus.cc:2932] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:02.540760 27108 catalog_manager.cc:5484] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:02.553478 30028 consensus_queue.cc:237] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:02.554119 27116 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240502 08:18:02.557644 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): Couldn't send request to peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26545a868bd6415c9c9db4869b2841c7. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:02.557900 30288 raft_consensus.cc:1261] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:02.558619 30158 raft_consensus.cc:1261] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:02.558931 31777 consensus_queue.cc:1035] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:02.559259 31912 consensus_queue.cc:1035] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:02.563494 30400 tablet_service.cc:1467] Processing CreateTablet for tablet a7fc48f77b044ce8bc9e74ecc0ef1d83 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0223e428283e4982928c2c5a2e70edf5]), partition=
I20240502 08:18:02.563783 30008 tablet_service.cc:1467] Processing CreateTablet for tablet a7fc48f77b044ce8bc9e74ecc0ef1d83 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0223e428283e4982928c2c5a2e70edf5]), partition=
I20240502 08:18:02.563858 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fc48f77b044ce8bc9e74ecc0ef1d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:02.564128 30008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fc48f77b044ce8bc9e74ecc0ef1d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:02.566243 31777 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:02.568828 27631 tablet_service.cc:1467] Processing CreateTablet for tablet a7fc48f77b044ce8bc9e74ecc0ef1d83 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0223e428283e4982928c2c5a2e70edf5]), partition=
I20240502 08:18:02.569088 31915 tablet_bootstrap.cc:492] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:02.569216 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fc48f77b044ce8bc9e74ecc0ef1d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:02.570425 31915 tablet_bootstrap.cc:654] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:02.572819 30288 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:02.579414 30158 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:02.581804 27116 catalog_manager.cc:5484] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:02.596509 31919 tablet_bootstrap.cc:492] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:02.599493 31919 tablet_bootstrap.cc:654] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:02.600145 31917 tablet_bootstrap.cc:492] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:02.605041 31917 tablet_bootstrap.cc:654] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:02.609390 31915 tablet_bootstrap.cc:492] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:02.609439 31919 tablet_bootstrap.cc:492] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:02.609598 31915 ts_tablet_manager.cc:1367] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.010s
I20240502 08:18:02.609714 31919 ts_tablet_manager.cc:1367] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240502 08:18:02.610855 31919 raft_consensus.cc:348] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:02.612085 31915 raft_consensus.cc:348] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:02.612258 31919 raft_consensus.cc:374] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:02.612421 31919 raft_consensus.cc:729] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:02.612509 31915 raft_consensus.cc:374] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:02.612751 31915 raft_consensus.cc:729] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:02.612746 31919 consensus_queue.cc:260] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:02.613057 31915 consensus_queue.cc:260] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:02.615872 31915 ts_tablet_manager.cc:1398] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:18:02.618882 31919 ts_tablet_manager.cc:1398] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 08:18:02.629251 31917 tablet_bootstrap.cc:492] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:02.629472 31917 ts_tablet_manager.cc:1367] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20240502 08:18:02.630414 31917 raft_consensus.cc:348] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:02.630684 31917 raft_consensus.cc:374] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:02.630800 31917 raft_consensus.cc:729] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:02.631086 31917 consensus_queue.cc:260] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:02.631738 31917 ts_tablet_manager.cc:1398] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:02.646912 30421 consensus_queue.cc:237] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
W20240502 08:18:02.652724 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 201d79895c2741dd9d9de0cc9f605a91. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:02.653055 27648 raft_consensus.cc:1261] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:02.653559 31309 consensus_queue.cc:1035] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:02.654229 30158 raft_consensus.cc:1261] T 201d79895c2741dd9d9de0cc9f605a91 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:02.654753 31872 consensus_queue.cc:1035] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:02.658975 31309 raft_consensus.cc:2932] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:02.663787 27110 catalog_manager.cc:5484] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:02.664240 27648 raft_consensus.cc:2932] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:02.669461 30158 raft_consensus.cc:2932] T 201d79895c2741dd9d9de0cc9f605a91 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:02.687330 27520 consensus_queue.cc:237] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:02.689265 27387 raft_consensus.cc:1261] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:02.689304 27258 raft_consensus.cc:1261] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:18:02.689569 27453 consensus_peers.cc:485] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 -> Peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Couldn't send request to peer 1f0bf4bc3c5146539146b825b048bff3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0bea60d096f9408c9f4a857f8824d6a2. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:02.689863 31359 consensus_queue.cc:1035] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:02.690224 31705 consensus_queue.cc:1035] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:02.692870 31719 raft_consensus.cc:2932] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:02.693514 27387 raft_consensus.cc:2932] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:02.694276 27258 raft_consensus.cc:2932] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:02.698917 27110 catalog_manager.cc:5484] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:02.701588 31938 ts_tablet_manager.cc:902] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Initiating tablet copy from peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:02.704674 31682 consensus_queue.cc:1035] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:02.707408 31309 raft_consensus.cc:482] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:02.707569 31309 raft_consensus.cc:504] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:02.708247 31309 leader_election.cc:288] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:02.708433 31666 consensus_queue.cc:1035] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:02.709051 30026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fc48f77b044ce8bc9e74ecc0ef1d83" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:02.709264 30026 raft_consensus.cc:2454] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:02.709692 30355 leader_election.cc:302] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters:
I20240502 08:18:02.711365 31872 raft_consensus.cc:2781] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:02.711495 31872 raft_consensus.cc:482] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:02.711583 31872 raft_consensus.cc:3037] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:02.712661 27648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fc48f77b044ce8bc9e74ecc0ef1d83" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:02.712896 27648 raft_consensus.cc:2454] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:02.716914 31872 raft_consensus.cc:504] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:02.717795 30026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fc48f77b044ce8bc9e74ecc0ef1d83" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:02.717825 27648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fc48f77b044ce8bc9e74ecc0ef1d83" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:02.717988 27648 raft_consensus.cc:3037] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:02.718035 30026 raft_consensus.cc:3037] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:02.720782 30026 raft_consensus.cc:2454] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:02.720829 27648 raft_consensus.cc:2454] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:02.721189 30358 leader_election.cc:302] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:02.721448 31309 raft_consensus.cc:2781] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:02.721606 31309 raft_consensus.cc:686] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:18:02.721800 31872 leader_election.cc:288] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:02.721930 31309 consensus_queue.cc:237] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:02.746919 27116 catalog_manager.cc:5484] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:02.761324 27110 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:02.761647 31719 consensus_queue.cc:1035] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:02.771585 31949 ts_tablet_manager.cc:902] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:02.776849 30400 tablet_service.cc:1467] Processing CreateTablet for tablet f1f5650965e841ad98d857ca32299037 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=aff097c8fa414c8aa9c675914ae4280a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:02.776968 30140 tablet_service.cc:1467] Processing CreateTablet for tablet f1f5650965e841ad98d857ca32299037 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=aff097c8fa414c8aa9c675914ae4280a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:02.777340 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f5650965e841ad98d857ca32299037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:02.777355 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f5650965e841ad98d857ca32299037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:02.782558 30401 tablet_service.cc:1467] Processing CreateTablet for tablet 2066c7aa052342a1aca3eb849aa251fc (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=aff097c8fa414c8aa9c675914ae4280a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:02.782951 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2066c7aa052342a1aca3eb849aa251fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:02.782852 30270 tablet_service.cc:1467] Processing CreateTablet for tablet f1f5650965e841ad98d857ca32299037 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=aff097c8fa414c8aa9c675914ae4280a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:02.782884 30269 tablet_service.cc:1467] Processing CreateTablet for tablet 2066c7aa052342a1aca3eb849aa251fc (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=aff097c8fa414c8aa9c675914ae4280a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:02.783358 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f5650965e841ad98d857ca32299037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:02.783994 30269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2066c7aa052342a1aca3eb849aa251fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:02.786473 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 16b897307b4c4cbea72f814f64f1e7d7 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=aff097c8fa414c8aa9c675914ae4280a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:02.786834 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b897307b4c4cbea72f814f64f1e7d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:02.788791 30138 tablet_service.cc:1467] Processing CreateTablet for tablet 16b897307b4c4cbea72f814f64f1e7d7 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=aff097c8fa414c8aa9c675914ae4280a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:02.789160 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b897307b4c4cbea72f814f64f1e7d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:02.790927 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 16b897307b4c4cbea72f814f64f1e7d7 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=aff097c8fa414c8aa9c675914ae4280a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:02.791375 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b897307b4c4cbea72f814f64f1e7d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:02.792996 31915 tablet_bootstrap.cc:492] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:02.794663 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 2066c7aa052342a1aca3eb849aa251fc (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=aff097c8fa414c8aa9c675914ae4280a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:02.795135 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2066c7aa052342a1aca3eb849aa251fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:02.797457 31915 tablet_bootstrap.cc:654] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:02.800693 31719 consensus_queue.cc:1035] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:02.808460 31962 tablet_bootstrap.cc:492] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:02.809541 31919 tablet_bootstrap.cc:492] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:02.811165 31919 tablet_bootstrap.cc:654] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:02.821686 31966 ts_tablet_manager.cc:902] T af2f11471c064f48924191fc2607d31c P 84cda20738d2452da166e31fc6da57a6: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:02.826200 31938 tablet_copy_client.cc:308] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.2:46291
I20240502 08:18:02.827842 27410 tablet_copy_service.cc:140] P 83e171a5e16c4008923f1d43fc5ce7b1: Received BeginTabletCopySession request for tablet 04a0a58eb52f41bf818e5102996d09e0 from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad ({username='slave'} at 127.20.207.8:58351)
I20240502 08:18:02.828052 27410 tablet_copy_service.cc:161] P 83e171a5e16c4008923f1d43fc5ce7b1: Beginning new tablet copy session on tablet 04a0a58eb52f41bf818e5102996d09e0 from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad at {username='slave'} at 127.20.207.8:58351: session id = 31a1e33dcdbf42f58ea72ce1c0afb0ad-04a0a58eb52f41bf818e5102996d09e0
I20240502 08:18:02.832013 27410 tablet_copy_source_session.cc:205] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:02.832805 31968 tablet_bootstrap.cc:492] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:02.833453 31938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a0a58eb52f41bf818e5102996d09e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:02.834079 31919 tablet_bootstrap.cc:492] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:02.834350 31919 ts_tablet_manager.cc:1367] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.002s
I20240502 08:18:02.835387 31919 raft_consensus.cc:348] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:02.835918 31919 raft_consensus.cc:374] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:02.836385 31919 raft_consensus.cc:729] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:02.836786 31963 tablet_bootstrap.cc:492] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:02.836740 31919 consensus_queue.cc:260] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:02.838595 31919 ts_tablet_manager.cc:1398] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:02.840550 31965 ts_tablet_manager.cc:902] T 83993ff2073842988937597dd3999d0d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Initiating tablet copy from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:02.845907 31968 tablet_bootstrap.cc:654] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:02.851898 31968 tablet_bootstrap.cc:492] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:02.854874 31968 ts_tablet_manager.cc:1367] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20240502 08:18:02.855922 31968 raft_consensus.cc:348] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:02.859158 31947 ts_tablet_manager.cc:902] T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:02.859257 31966 tablet_copy_client.cc:308] T af2f11471c064f48924191fc2607d31c P 84cda20738d2452da166e31fc6da57a6: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:18:02.861024 31968 raft_consensus.cc:374] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:02.861131 31968 raft_consensus.cc:729] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:02.861379 31968 consensus_queue.cc:260] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:02.861513 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:02.861856 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet af2f11471c064f48924191fc2607d31c from peer 84cda20738d2452da166e31fc6da57a6 ({username='slave'} at 127.20.207.6:52973)
I20240502 08:18:02.862053 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet af2f11471c064f48924191fc2607d31c from peer 84cda20738d2452da166e31fc6da57a6 at {username='slave'} at 127.20.207.6:52973: session id = 84cda20738d2452da166e31fc6da57a6-af2f11471c064f48924191fc2607d31c
I20240502 08:18:02.864471 31968 ts_tablet_manager.cc:1398] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 08:18:02.865110 27278 tablet_copy_source_session.cc:205] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:02.865767 31938 tablet_copy_client.cc:783] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:02.866204 31938 tablet_copy_client.cc:647] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:02.868510 27648 raft_consensus.cc:1261] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:02.869066 31938 tablet_copy_client.cc:523] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:02.870069 31864 consensus_queue.cc:1035] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:02.870965 31966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2f11471c064f48924191fc2607d31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:02.876781 30028 raft_consensus.cc:1261] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:02.884759 31966 tablet_copy_client.cc:783] T af2f11471c064f48924191fc2607d31c P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:02.885190 31966 tablet_copy_client.cc:647] T af2f11471c064f48924191fc2607d31c P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:02.885752 27108 catalog_manager.cc:5576] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309) does not have the latest schema for tablet a7fc48f77b044ce8bc9e74ecc0ef1d83 (table rebalancer_dynamic_table [id=0223e428283e4982928c2c5a2e70edf5]). Expected version 1 got 0
I20240502 08:18:02.877393 31864 consensus_queue.cc:1035] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20240502 08:18:02.889829 31966 tablet_copy_client.cc:523] T af2f11471c064f48924191fc2607d31c P 84cda20738d2452da166e31fc6da57a6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:02.893368 31966 tablet_bootstrap.cc:492] T af2f11471c064f48924191fc2607d31c P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:02.893810 31985 tablet.cc:1722] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:02.889333 31983 tablet.cc:1722] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:02.896615 31983 tablet.cc:1620] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:02.896378 31753 raft_consensus.cc:482] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:02.896819 31753 raft_consensus.cc:504] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:02.897720 31753 leader_election.cc:288] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:02.899010 31985 tablet.cc:1620] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442: MemRowSet was empty: no flush needed.
I20240502 08:18:02.899221 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b897307b4c4cbea72f814f64f1e7d7" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:02.899605 31965 tablet_copy_client.cc:308] T 83993ff2073842988937597dd3999d0d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.8:41619
I20240502 08:18:02.898159 30156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b897307b4c4cbea72f814f64f1e7d7" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:02.901325 30443 tablet_copy_service.cc:140] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Received BeginTabletCopySession request for tablet 83993ff2073842988937597dd3999d0d from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 ({username='slave'} at 127.20.207.1:41297)
I20240502 08:18:02.901578 30443 tablet_copy_service.cc:161] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Beginning new tablet copy session on tablet 83993ff2073842988937597dd3999d0d from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 at {username='slave'} at 127.20.207.1:41297: session id = 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-83993ff2073842988937597dd3999d0d
W20240502 08:18:02.899924 27584 leader_election.cc:341] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:18:02.901947 27587 leader_election.cc:341] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:02.902352 27587 leader_election.cc:302] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:02.902891 31753 raft_consensus.cc:2726] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:02.903285 31680 raft_consensus.cc:482] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:02.903412 31680 raft_consensus.cc:504] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:02.904018 31680 leader_election.cc:288] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:02.904350 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2066c7aa052342a1aca3eb849aa251fc" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
W20240502 08:18:02.904822 27191 leader_election.cc:341] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:02.904927 30443 tablet_copy_source_session.cc:205] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:02.876853 31938 tablet_bootstrap.cc:492] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:02.906836 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2066c7aa052342a1aca3eb849aa251fc" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:02.907097 27116 catalog_manager.cc:5576] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309) does not have the latest schema for tablet a7fc48f77b044ce8bc9e74ecc0ef1d83 (table rebalancer_dynamic_table [id=0223e428283e4982928c2c5a2e70edf5]). Expected version 1 got 0
W20240502 08:18:02.907708 27191 leader_election.cc:341] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:02.907855 27191 leader_election.cc:302] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:02.908290 31680 raft_consensus.cc:2726] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:02.909392 31965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83993ff2073842988937597dd3999d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:02.930085 31987 ts_tablet_manager.cc:902] T e67c99e57a73443da13527ed33326a4e P b01e7c8a302d41078fa39cfef665ff59: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:02.930290 27116 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet a7fc48f77b044ce8bc9e74ecc0ef1d83 (table rebalancer_dynamic_table [id=0223e428283e4982928c2c5a2e70edf5]). Expected version 1 got 0
I20240502 08:18:02.930353 31965 tablet_copy_client.cc:783] T 83993ff2073842988937597dd3999d0d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:02.930799 31965 tablet_copy_client.cc:647] T 83993ff2073842988937597dd3999d0d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:02.932766 31965 tablet_copy_client.cc:523] T 83993ff2073842988937597dd3999d0d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:02.934185 27110 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet a7fc48f77b044ce8bc9e74ecc0ef1d83 (table rebalancer_dynamic_table [id=0223e428283e4982928c2c5a2e70edf5]). Expected version 1 got 0
I20240502 08:18:02.935081 27108 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=0223e428283e4982928c2c5a2e70edf5] alter complete (version 1)
I20240502 08:18:02.935418 31988 ts_tablet_manager.cc:902] T a6dce110666a4a038f5f46d48a3ec7af P 49c5c8f70e494edd9fcda40cc0af6cc4: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:02.945278 31965 tablet_bootstrap.cc:492] T 83993ff2073842988937597dd3999d0d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:02.952364 31963 tablet_bootstrap.cc:654] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:02.954049 31962 tablet_bootstrap.cc:654] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:02.959934 31991 tablet.cc:1722] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:02.961355 31949 tablet_copy_client.cc:308] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:02.964526 27110 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet a7fc48f77b044ce8bc9e74ecc0ef1d83 (table rebalancer_dynamic_table [id=0223e428283e4982928c2c5a2e70edf5]). Expected version 1 got 0
I20240502 08:18:02.965914 31947 tablet_copy_client.cc:308] T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:18:02.966398 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet 09aa1b5d99114b3095be08c9547fbb54 from peer 49c5c8f70e494edd9fcda40cc0af6cc4 ({username='slave'} at 127.20.207.7:58763)
I20240502 08:18:02.966563 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet 09aa1b5d99114b3095be08c9547fbb54 from peer 49c5c8f70e494edd9fcda40cc0af6cc4 at {username='slave'} at 127.20.207.7:58763: session id = 49c5c8f70e494edd9fcda40cc0af6cc4-09aa1b5d99114b3095be08c9547fbb54
I20240502 08:18:02.966620 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet 6978aa3955504c548778066ddd0910e5 from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad ({username='slave'} at 127.20.207.8:40413)
I20240502 08:18:02.966769 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet 6978aa3955504c548778066ddd0910e5 from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad at {username='slave'} at 127.20.207.8:40413: session id = 31a1e33dcdbf42f58ea72ce1c0afb0ad-6978aa3955504c548778066ddd0910e5
I20240502 08:18:02.967316 31963 tablet_bootstrap.cc:492] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:02.967454 31963 ts_tablet_manager.cc:1367] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.131s user 0.004s sys 0.000s
I20240502 08:18:02.968420 31991 tablet.cc:1620] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:02.969444 27541 tablet_copy_source_session.cc:205] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:02.969462 31963 raft_consensus.cc:348] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:02.969594 27278 tablet_copy_source_session.cc:205] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:02.971060 31949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09aa1b5d99114b3095be08c9547fbb54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:02.971599 31947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6978aa3955504c548778066ddd0910e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:02.974936 31963 raft_consensus.cc:374] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:02.975167 31963 raft_consensus.cc:729] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:02.975562 31963 consensus_queue.cc:260] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:02.978940 31962 tablet_bootstrap.cc:492] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:02.979851 31963 ts_tablet_manager.cc:1398] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240502 08:18:02.980818 31962 ts_tablet_manager.cc:1367] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.172s user 0.005s sys 0.000s
I20240502 08:18:02.986280 31963 tablet_bootstrap.cc:492] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:02.987659 31988 tablet_copy_client.cc:308] T a6dce110666a4a038f5f46d48a3ec7af P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:02.988399 31915 tablet_bootstrap.cc:492] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:02.988440 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet a6dce110666a4a038f5f46d48a3ec7af from peer 49c5c8f70e494edd9fcda40cc0af6cc4 ({username='slave'} at 127.20.207.7:58763)
I20240502 08:18:02.988581 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet a6dce110666a4a038f5f46d48a3ec7af from peer 49c5c8f70e494edd9fcda40cc0af6cc4 at {username='slave'} at 127.20.207.7:58763: session id = 49c5c8f70e494edd9fcda40cc0af6cc4-a6dce110666a4a038f5f46d48a3ec7af
I20240502 08:18:02.988610 31915 ts_tablet_manager.cc:1367] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.196s user 0.006s sys 0.010s
I20240502 08:18:02.989365 31915 raft_consensus.cc:348] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:02.989606 31915 raft_consensus.cc:374] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:02.989673 31915 raft_consensus.cc:729] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:02.989934 31915 consensus_queue.cc:260] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:02.991398 27541 tablet_copy_source_session.cc:205] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:02.992683 31962 raft_consensus.cc:348] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:02.992836 31988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6dce110666a4a038f5f46d48a3ec7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:02.997767 31915 ts_tablet_manager.cc:1398] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 08:18:03.001127 31915 tablet_bootstrap.cc:492] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:03.001855 31988 tablet_copy_client.cc:783] T a6dce110666a4a038f5f46d48a3ec7af P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:02.993284 31949 tablet_copy_client.cc:783] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:03.002319 31988 tablet_copy_client.cc:647] T a6dce110666a4a038f5f46d48a3ec7af P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:03.002456 31949 tablet_copy_client.cc:647] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:03.002601 31962 raft_consensus.cc:374] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:03.003626 31915 tablet_bootstrap.cc:654] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.004400 31988 tablet_copy_client.cc:523] T a6dce110666a4a038f5f46d48a3ec7af P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:03.006970 31965 tablet_bootstrap.cc:492] T 83993ff2073842988937597dd3999d0d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:03.007536 31988 tablet_bootstrap.cc:492] T a6dce110666a4a038f5f46d48a3ec7af P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:03.009193 31949 tablet_copy_client.cc:523] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:03.009941 31965 tablet_bootstrap.cc:492] T 83993ff2073842988937597dd3999d0d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap complete.
I20240502 08:18:03.010308 31965 ts_tablet_manager.cc:1367] T 83993ff2073842988937597dd3999d0d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.002s
I20240502 08:18:03.011732 31965 raft_consensus.cc:348] T 83993ff2073842988937597dd3999d0d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:03.012084 31965 raft_consensus.cc:729] T 83993ff2073842988937597dd3999d0d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: LEARNER
I20240502 08:18:03.012384 31965 consensus_queue.cc:260] T 83993ff2073842988937597dd3999d0d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:03.012876 31949 tablet_bootstrap.cc:492] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:03.026398 31965 ts_tablet_manager.cc:1398] T 83993ff2073842988937597dd3999d0d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240502 08:18:03.030258 32008 ts_tablet_manager.cc:902] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:03.004256 31962 raft_consensus.cc:729] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:03.031095 31962 consensus_queue.cc:260] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:03.033463 31962 ts_tablet_manager.cc:1398] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20240502 08:18:03.033761 31962 tablet_bootstrap.cc:492] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:03.035929 31987 tablet_copy_client.cc:308] T e67c99e57a73443da13527ed33326a4e P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:18:03.037308 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet e67c99e57a73443da13527ed33326a4e from peer b01e7c8a302d41078fa39cfef665ff59 ({username='slave'} at 127.20.207.3:51699)
I20240502 08:18:03.037519 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet e67c99e57a73443da13527ed33326a4e from peer b01e7c8a302d41078fa39cfef665ff59 at {username='slave'} at 127.20.207.3:51699: session id = b01e7c8a302d41078fa39cfef665ff59-e67c99e57a73443da13527ed33326a4e
I20240502 08:18:03.043088 27278 tablet_copy_source_session.cc:205] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:03.053527 32008 tablet_copy_client.cc:308] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:18:03.054586 32013 ts_tablet_manager.cc:902] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Initiating tablet copy from peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:03.057261 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet dd887cbb35c74e54a43d3dc6a2dde817 from peer 1f0bf4bc3c5146539146b825b048bff3 ({username='slave'} at 127.20.207.5:51633)
I20240502 08:18:03.057421 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet dd887cbb35c74e54a43d3dc6a2dde817 from peer 1f0bf4bc3c5146539146b825b048bff3 at {username='slave'} at 127.20.207.5:51633: session id = 1f0bf4bc3c5146539146b825b048bff3-dd887cbb35c74e54a43d3dc6a2dde817
I20240502 08:18:03.057350 27108 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:03.060489 27278 tablet_copy_source_session.cc:205] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:03.060690 31963 tablet_bootstrap.cc:654] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.061674 32008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd887cbb35c74e54a43d3dc6a2dde817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.027691 30443 tablet_copy_service.cc:342] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Request end of tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-83993ff2073842988937597dd3999d0d received from {username='slave'} at 127.20.207.1:41297
I20240502 08:18:03.064194 30443 tablet_copy_service.cc:433] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: ending tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-83993ff2073842988937597dd3999d0d on tablet 83993ff2073842988937597dd3999d0d with peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60
I20240502 08:18:03.070384 31987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e67c99e57a73443da13527ed33326a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.076393 31983 tablet.cc:1722] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:03.076844 31983 tablet.cc:1620] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:03.076951 31991 tablet.cc:1722] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:03.077402 31991 tablet.cc:1620] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:03.078259 31985 tablet.cc:1722] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:03.078719 31985 tablet.cc:1620] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442: MemRowSet was empty: no flush needed.
I20240502 08:18:03.086129 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=0223e428283e4982928c2c5a2e70edf5] alter complete (version 2)
I20240502 08:18:03.090021 31962 tablet_bootstrap.cc:654] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.090095 32008 tablet_copy_client.cc:783] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:03.090504 32008 tablet_copy_client.cc:647] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:03.093097 32007 ts_tablet_manager.cc:902] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Initiating tablet copy from peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:03.093745 32008 tablet_copy_client.cc:523] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:03.099972 32008 tablet_bootstrap.cc:492] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:03.109275 31920 raft_consensus.cc:482] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:03.109464 31920 raft_consensus.cc:504] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:03.110391 31920 leader_election.cc:288] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:03.110989 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2066c7aa052342a1aca3eb849aa251fc" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:03.111213 27258 raft_consensus.cc:2454] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:03.111392 31947 tablet_copy_client.cc:783] T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:03.113016 31947 tablet_copy_client.cc:647] T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:03.114879 30225 leader_election.cc:302] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:18:03.116569 31920 raft_consensus.cc:2781] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:03.116708 31920 raft_consensus.cc:482] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:03.116799 31920 raft_consensus.cc:3037] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.117986 31947 tablet_copy_client.cc:523] T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:03.118340 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2066c7aa052342a1aca3eb849aa251fc" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:03.120690 31920 raft_consensus.cc:504] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:03.120942 30421 raft_consensus.cc:2454] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:03.123227 31920 leader_election.cc:288] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:03.132655 31987 tablet_copy_client.cc:783] T e67c99e57a73443da13527ed33326a4e P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:03.138420 31987 tablet_copy_client.cc:647] T e67c99e57a73443da13527ed33326a4e P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:03.139560 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2066c7aa052342a1aca3eb849aa251fc" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:03.139770 30423 raft_consensus.cc:3037] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.143496 30423 raft_consensus.cc:2454] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:03.144003 30223 leader_election.cc:302] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:18:03.144380 31920 raft_consensus.cc:2781] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:03.144533 31920 raft_consensus.cc:686] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:18:03.144826 31920 consensus_queue.cc:237] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:03.146962 31963 tablet_bootstrap.cc:492] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:03.147186 31963 ts_tablet_manager.cc:1367] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.161s user 0.008s sys 0.000s
I20240502 08:18:03.148056 31963 raft_consensus.cc:348] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:03.152305 31915 tablet_bootstrap.cc:492] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:03.152493 31915 ts_tablet_manager.cc:1367] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.151s user 0.006s sys 0.004s
I20240502 08:18:03.153992 31963 raft_consensus.cc:374] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:03.154457 31963 raft_consensus.cc:729] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:03.154645 31915 raft_consensus.cc:348] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:03.155076 31963 consensus_queue.cc:260] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:03.162241 31963 ts_tablet_manager.cc:1398] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 08:18:03.163224 27110 catalog_manager.cc:5484] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7). New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:03.171432 31962 tablet_bootstrap.cc:492] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:03.171662 31962 ts_tablet_manager.cc:1367] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.138s user 0.004s sys 0.004s
I20240502 08:18:03.172523 32008 tablet_bootstrap.cc:492] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:03.173017 32008 tablet_bootstrap.cc:492] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap complete.
I20240502 08:18:03.173374 32008 ts_tablet_manager.cc:1367] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.073s user 0.051s sys 0.005s
I20240502 08:18:03.174165 32008 raft_consensus.cc:348] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:03.174436 32008 raft_consensus.cc:729] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: LEARNER
I20240502 08:18:03.174626 32008 consensus_queue.cc:260] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:03.175223 32008 ts_tablet_manager.cc:1398] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:03.176379 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session 1f0bf4bc3c5146539146b825b048bff3-dd887cbb35c74e54a43d3dc6a2dde817 received from {username='slave'} at 127.20.207.5:51633
I20240502 08:18:03.176501 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session 1f0bf4bc3c5146539146b825b048bff3-dd887cbb35c74e54a43d3dc6a2dde817 on tablet dd887cbb35c74e54a43d3dc6a2dde817 with peer 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:18:03.176628 31915 raft_consensus.cc:374] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:03.177049 31915 raft_consensus.cc:729] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:03.178236 31962 raft_consensus.cc:348] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:03.182160 31987 tablet_copy_client.cc:523] T e67c99e57a73443da13527ed33326a4e P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:03.182337 31915 consensus_queue.cc:260] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:03.182945 31915 ts_tablet_manager.cc:1398] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.029s user 0.001s sys 0.000s
I20240502 08:18:03.183573 31962 raft_consensus.cc:374] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:03.183651 31962 raft_consensus.cc:729] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:03.183883 31962 consensus_queue.cc:260] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:03.184422 31962 ts_tablet_manager.cc:1398] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240502 08:18:03.184592 32013 tablet_copy_client.cc:308] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.5:37815
I20240502 08:18:03.184741 31962 tablet_bootstrap.cc:492] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:03.185264 30048 tablet_copy_service.cc:140] P 1f0bf4bc3c5146539146b825b048bff3: Received BeginTabletCopySession request for tablet 26545a868bd6415c9c9db4869b2841c7 from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 ({username='slave'} at 127.20.207.1:49229)
I20240502 08:18:03.185424 30048 tablet_copy_service.cc:161] P 1f0bf4bc3c5146539146b825b048bff3: Beginning new tablet copy session on tablet 26545a868bd6415c9c9db4869b2841c7 from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 at {username='slave'} at 127.20.207.1:49229: session id = 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-26545a868bd6415c9c9db4869b2841c7
I20240502 08:18:03.186408 31962 tablet_bootstrap.cc:654] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
W20240502 08:18:03.183328 30225 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.207.1:38505, user_credentials={real_user=slave}} blocked reactor thread for 68438.6us
I20240502 08:18:03.162868 32007 tablet_copy_client.cc:308] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.4:33309
I20240502 08:18:03.188328 30048 tablet_copy_source_session.cc:205] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:03.192365 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2066c7aa052342a1aca3eb849aa251fc" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:03.192572 27257 raft_consensus.cc:3037] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.193444 27108 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:03.196012 32013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26545a868bd6415c9c9db4869b2841c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.196576 27257 raft_consensus.cc:2454] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:03.206694 27671 tablet_copy_service.cc:140] P bf8ac71b15f740a0a928911360773442: Received BeginTabletCopySession request for tablet cad036d297bd45d7b7c5a1f19356906f from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad ({username='slave'} at 127.20.207.8:38973)
I20240502 08:18:03.206867 27671 tablet_copy_service.cc:161] P bf8ac71b15f740a0a928911360773442: Beginning new tablet copy session on tablet cad036d297bd45d7b7c5a1f19356906f from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad at {username='slave'} at 127.20.207.8:38973: session id = 31a1e33dcdbf42f58ea72ce1c0afb0ad-cad036d297bd45d7b7c5a1f19356906f
I20240502 08:18:03.210706 27671 tablet_copy_source_session.cc:205] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:03.212419 32007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cad036d297bd45d7b7c5a1f19356906f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.168107 31947 tablet_bootstrap.cc:492] T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:03.217376 32013 tablet_copy_client.cc:783] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:03.217666 32013 tablet_copy_client.cc:647] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:03.218523 31949 tablet_bootstrap.cc:492] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:03.220535 32013 tablet_copy_client.cc:523] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:03.218958 31949 tablet_bootstrap.cc:492] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap complete.
I20240502 08:18:03.225127 31949 ts_tablet_manager.cc:1367] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.212s user 0.049s sys 0.007s
I20240502 08:18:03.225883 31949 raft_consensus.cc:348] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:03.226186 31949 raft_consensus.cc:729] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: LEARNER
I20240502 08:18:03.226409 31949 consensus_queue.cc:260] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:03.232193 31949 ts_tablet_manager.cc:1398] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:18:03.233264 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-09aa1b5d99114b3095be08c9547fbb54 received from {username='slave'} at 127.20.207.7:58763
I20240502 08:18:03.233409 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-09aa1b5d99114b3095be08c9547fbb54 on tablet 09aa1b5d99114b3095be08c9547fbb54 with peer 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:03.233631 32013 tablet_bootstrap.cc:492] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:03.235814 31983 tablet.cc:1722] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:03.236838 32007 tablet_copy_client.cc:783] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:03.237118 32007 tablet_copy_client.cc:647] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:03.241690 31985 tablet.cc:1722] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:03.245772 27108 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=0223e428283e4982928c2c5a2e70edf5] alter complete (version 3)
I20240502 08:18:03.248042 32007 tablet_copy_client.cc:523] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:03.251534 32007 tablet_bootstrap.cc:492] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:03.252460 32013 tablet_bootstrap.cc:492] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:03.252898 32013 tablet_bootstrap.cc:492] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap complete.
I20240502 08:18:03.253187 32013 ts_tablet_manager.cc:1367] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.004s
I20240502 08:18:03.253455 32042 ts_tablet_manager.cc:902] T 201d79895c2741dd9d9de0cc9f605a91 P 83e171a5e16c4008923f1d43fc5ce7b1: Initiating tablet copy from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
W20240502 08:18:03.254312 29961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.7:40343, user_credentials={real_user=slave}} blocked reactor thread for 63498.7us
I20240502 08:18:03.256862 32013 raft_consensus.cc:348] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:03.257064 32008 ts_tablet_manager.cc:902] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:03.257115 32013 raft_consensus.cc:729] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: LEARNER
I20240502 08:18:03.257369 32013 consensus_queue.cc:260] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:03.259819 31991 tablet.cc:1722] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:03.270660 31987 tablet_bootstrap.cc:492] T e67c99e57a73443da13527ed33326a4e P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:03.270787 27257 raft_consensus.cc:1203] T 83993ff2073842988937597dd3999d0d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:03.273957 32013 ts_tablet_manager.cc:1398] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20240502 08:18:03.274693 30048 tablet_copy_service.cc:342] P 1f0bf4bc3c5146539146b825b048bff3: Request end of tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-26545a868bd6415c9c9db4869b2841c7 received from {username='slave'} at 127.20.207.1:49229
I20240502 08:18:03.274811 30048 tablet_copy_service.cc:433] P 1f0bf4bc3c5146539146b825b048bff3: ending tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-26545a868bd6415c9c9db4869b2841c7 on tablet 26545a868bd6415c9c9db4869b2841c7 with peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60
I20240502 08:18:03.280428 31938 tablet_bootstrap.cc:492] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:03.281011 31938 tablet_bootstrap.cc:492] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap complete.
I20240502 08:18:03.281267 31938 ts_tablet_manager.cc:1367] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.405s user 0.042s sys 0.016s
I20240502 08:18:03.287703 31962 tablet_bootstrap.cc:492] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:03.287891 31962 ts_tablet_manager.cc:1367] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.103s user 0.004s sys 0.003s
I20240502 08:18:03.288751 31962 raft_consensus.cc:348] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:03.288969 31962 raft_consensus.cc:374] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:03.289057 31962 raft_consensus.cc:729] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:03.289278 31962 consensus_queue.cc:260] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:03.289906 31962 ts_tablet_manager.cc:1398] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:03.284308 31938 raft_consensus.cc:348] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:03.299602 31938 raft_consensus.cc:729] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: LEARNER
I20240502 08:18:03.305682 31988 tablet_bootstrap.cc:492] T a6dce110666a4a038f5f46d48a3ec7af P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:03.299861 31938 consensus_queue.cc:260] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:03.308065 31988 tablet_bootstrap.cc:492] T a6dce110666a4a038f5f46d48a3ec7af P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap complete.
I20240502 08:18:03.308388 32008 tablet_copy_client.cc:308] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:03.309074 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet 0bea60d096f9408c9f4a857f8824d6a2 from peer 1f0bf4bc3c5146539146b825b048bff3 ({username='slave'} at 127.20.207.5:54501)
I20240502 08:18:03.309290 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet 0bea60d096f9408c9f4a857f8824d6a2 from peer 1f0bf4bc3c5146539146b825b048bff3 at {username='slave'} at 127.20.207.5:54501: session id = 1f0bf4bc3c5146539146b825b048bff3-0bea60d096f9408c9f4a857f8824d6a2
I20240502 08:18:03.311738 31988 ts_tablet_manager.cc:1367] T a6dce110666a4a038f5f46d48a3ec7af P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.304s user 0.043s sys 0.007s
I20240502 08:18:03.312242 27541 tablet_copy_source_session.cc:205] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:03.313563 32008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bea60d096f9408c9f4a857f8824d6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.312470 31988 raft_consensus.cc:348] T a6dce110666a4a038f5f46d48a3ec7af P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:03.314438 31988 raft_consensus.cc:729] T a6dce110666a4a038f5f46d48a3ec7af P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: LEARNER
I20240502 08:18:03.314703 31988 consensus_queue.cc:260] T a6dce110666a4a038f5f46d48a3ec7af P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:03.315745 31938 ts_tablet_manager.cc:1398] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20240502 08:18:03.318713 31988 ts_tablet_manager.cc:1398] T a6dce110666a4a038f5f46d48a3ec7af P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:18:03.323999 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-a6dce110666a4a038f5f46d48a3ec7af received from {username='slave'} at 127.20.207.7:58763
I20240502 08:18:03.324126 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-a6dce110666a4a038f5f46d48a3ec7af on tablet a6dce110666a4a038f5f46d48a3ec7af with peer 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:03.324430 27410 tablet_copy_service.cc:342] P 83e171a5e16c4008923f1d43fc5ce7b1: Request end of tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-04a0a58eb52f41bf818e5102996d09e0 received from {username='slave'} at 127.20.207.8:58351
I20240502 08:18:03.324558 27410 tablet_copy_service.cc:433] P 83e171a5e16c4008923f1d43fc5ce7b1: ending tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-04a0a58eb52f41bf818e5102996d09e0 on tablet 04a0a58eb52f41bf818e5102996d09e0 with peer 31a1e33dcdbf42f58ea72ce1c0afb0ad
I20240502 08:18:03.326066 31966 tablet_bootstrap.cc:492] T af2f11471c064f48924191fc2607d31c P 84cda20738d2452da166e31fc6da57a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:03.326635 31966 tablet_bootstrap.cc:492] T af2f11471c064f48924191fc2607d31c P 84cda20738d2452da166e31fc6da57a6: Bootstrap complete.
I20240502 08:18:03.326974 31966 ts_tablet_manager.cc:1367] T af2f11471c064f48924191fc2607d31c P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.434s user 0.054s sys 0.004s
I20240502 08:18:03.330080 32042 tablet_copy_client.cc:308] T 201d79895c2741dd9d9de0cc9f605a91 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.8:41619
I20240502 08:18:03.330900 30443 tablet_copy_service.cc:140] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Received BeginTabletCopySession request for tablet 201d79895c2741dd9d9de0cc9f605a91 from peer 83e171a5e16c4008923f1d43fc5ce7b1 ({username='slave'} at 127.20.207.2:33397)
I20240502 08:18:03.331075 30443 tablet_copy_service.cc:161] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Beginning new tablet copy session on tablet 201d79895c2741dd9d9de0cc9f605a91 from peer 83e171a5e16c4008923f1d43fc5ce7b1 at {username='slave'} at 127.20.207.2:33397: session id = 83e171a5e16c4008923f1d43fc5ce7b1-201d79895c2741dd9d9de0cc9f605a91
I20240502 08:18:03.331431 32008 tablet_copy_client.cc:783] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:03.331642 32008 tablet_copy_client.cc:647] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:03.334173 31682 raft_consensus.cc:482] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:03.334326 31682 raft_consensus.cc:504] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:03.334949 31682 leader_election.cc:288] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:03.336015 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b897307b4c4cbea72f814f64f1e7d7" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:03.336264 30290 raft_consensus.cc:2454] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:03.336349 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b897307b4c4cbea72f814f64f1e7d7" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:03.336555 27650 raft_consensus.cc:2454] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:03.338454 30096 leader_election.cc:302] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:03.339012 31682 raft_consensus.cc:2781] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:03.339110 31682 raft_consensus.cc:482] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:03.339215 31682 raft_consensus.cc:3037] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.339838 31966 raft_consensus.cc:348] T af2f11471c064f48924191fc2607d31c P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:03.340104 31966 raft_consensus.cc:729] T af2f11471c064f48924191fc2607d31c P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: LEARNER
I20240502 08:18:03.341622 31966 consensus_queue.cc:260] T af2f11471c064f48924191fc2607d31c P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:03.342456 32008 tablet_copy_client.cc:523] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:03.346431 30443 tablet_copy_source_session.cc:205] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:03.341984 31682 raft_consensus.cc:504] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:03.349237 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b897307b4c4cbea72f814f64f1e7d7" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:03.349440 27650 raft_consensus.cc:3037] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.344427 31966 ts_tablet_manager.cc:1398] T af2f11471c064f48924191fc2607d31c P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240502 08:18:03.351507 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session 84cda20738d2452da166e31fc6da57a6-af2f11471c064f48924191fc2607d31c received from {username='slave'} at 127.20.207.6:52973
I20240502 08:18:03.351655 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session 84cda20738d2452da166e31fc6da57a6-af2f11471c064f48924191fc2607d31c on tablet af2f11471c064f48924191fc2607d31c with peer 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:03.354671 32008 tablet_bootstrap.cc:492] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:03.354871 27108 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:03.355057 27108 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:03.359894 27108 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet a7fc48f77b044ce8bc9e74ecc0ef1d83
I20240502 08:18:03.360860 30400 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fc48f77b044ce8bc9e74ecc0ef1d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:03 UTC) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:03.361230 30008 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fc48f77b044ce8bc9e74ecc0ef1d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:03 UTC) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:03.362289 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fc48f77b044ce8bc9e74ecc0ef1d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:03 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:18:03.364785 27650 raft_consensus.cc:2454] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:03.367625 30096 leader_election.cc:302] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:03.367884 31666 raft_consensus.cc:2781] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:03.372320 31920 raft_consensus.cc:482] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:03.372515 31920 raft_consensus.cc:504] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:03.373210 31920 leader_election.cc:288] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:03.373618 30156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f5650965e841ad98d857ca32299037" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:03.373839 30156 raft_consensus.cc:2454] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:03.374176 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f5650965e841ad98d857ca32299037" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:03.374289 30226 leader_election.cc:302] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:03.374399 30423 raft_consensus.cc:2454] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:03.375206 31920 raft_consensus.cc:2781] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:03.375316 31920 raft_consensus.cc:482] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:03.375433 31920 raft_consensus.cc:3037] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.376119 31682 leader_election.cc:288] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:03.376233 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b897307b4c4cbea72f814f64f1e7d7" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:03.376456 30290 raft_consensus.cc:3037] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.378474 31920 raft_consensus.cc:504] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:03.379213 30290 raft_consensus.cc:2454] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:03.379220 31920 leader_election.cc:288] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:03.379868 30156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f5650965e841ad98d857ca32299037" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:03.380069 30156 raft_consensus.cc:3037] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.380458 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f5650965e841ad98d857ca32299037" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:03.380630 30423 raft_consensus.cc:3037] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.382664 30156 raft_consensus.cc:2454] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:03.383077 30226 leader_election.cc:302] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:03.383216 31666 raft_consensus.cc:686] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:03.383355 30423 raft_consensus.cc:2454] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:03.383386 31920 raft_consensus.cc:2781] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:03.383584 31920 raft_consensus.cc:686] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:18:03.383823 31920 consensus_queue.cc:237] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:03.383486 31666 consensus_queue.cc:237] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:03.390605 32042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 201d79895c2741dd9d9de0cc9f605a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.400686 30156 raft_consensus.cc:1203] T af2f11471c064f48924191fc2607d31c P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 08:18:03.398252 31947 log.cc:924] Time spent T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20240502 08:18:03.407933 27108 catalog_manager.cc:5484] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7). New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:03.409303 27110 catalog_manager.cc:5484] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:03.411660 31987 tablet_bootstrap.cc:492] T e67c99e57a73443da13527ed33326a4e P b01e7c8a302d41078fa39cfef665ff59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:03.420773 31987 tablet_bootstrap.cc:492] T e67c99e57a73443da13527ed33326a4e P b01e7c8a302d41078fa39cfef665ff59: Bootstrap complete.
I20240502 08:18:03.421145 31987 ts_tablet_manager.cc:1367] T e67c99e57a73443da13527ed33326a4e P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.151s user 0.002s sys 0.006s
I20240502 08:18:03.427201 31987 raft_consensus.cc:348] T e67c99e57a73443da13527ed33326a4e P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:03.428736 31987 raft_consensus.cc:729] T e67c99e57a73443da13527ed33326a4e P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: LEARNER
I20240502 08:18:03.429603 31987 consensus_queue.cc:260] T e67c99e57a73443da13527ed33326a4e P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:03.433945 31987 ts_tablet_manager.cc:1398] T e67c99e57a73443da13527ed33326a4e P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 08:18:03.434139 32042 tablet_copy_client.cc:783] T 201d79895c2741dd9d9de0cc9f605a91 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:03.434458 32042 tablet_copy_client.cc:647] T 201d79895c2741dd9d9de0cc9f605a91 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:03.436466 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session b01e7c8a302d41078fa39cfef665ff59-e67c99e57a73443da13527ed33326a4e received from {username='slave'} at 127.20.207.3:51699
I20240502 08:18:03.436646 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session b01e7c8a302d41078fa39cfef665ff59-e67c99e57a73443da13527ed33326a4e on tablet e67c99e57a73443da13527ed33326a4e with peer b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:03.443971 32042 tablet_copy_client.cc:523] T 201d79895c2741dd9d9de0cc9f605a91 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:03.450428 32068 tablet_replica.cc:330] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:18:03.450747 30026 raft_consensus.cc:1203] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:03.450724 32067 tablet_replica.cc:330] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:03.452452 32067 raft_consensus.cc:2229] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:03.452677 32067 raft_consensus.cc:2258] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:03.451020 32068 raft_consensus.cc:2229] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:03.453544 32068 raft_consensus.cc:2258] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:03.454114 32066 tablet_replica.cc:330] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:03.454352 32066 raft_consensus.cc:2229] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:03.454697 32066 raft_consensus.cc:2258] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:03.455116 32067 ts_tablet_manager.cc:1875] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:03.458544 32068 ts_tablet_manager.cc:1875] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:03.459602 32042 tablet_bootstrap.cc:492] T 201d79895c2741dd9d9de0cc9f605a91 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:03.464632 27109 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:03.465188 32066 ts_tablet_manager.cc:1875] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:03.476423 30008 tablet_service.cc:1467] Processing CreateTablet for tablet cb66fb2351f442c4bd3137274cdb5449 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6336a199bdbc48389a3a983413491684]), partition=
I20240502 08:18:03.476667 30268 tablet_service.cc:1467] Processing CreateTablet for tablet cb66fb2351f442c4bd3137274cdb5449 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6336a199bdbc48389a3a983413491684]), partition=
I20240502 08:18:03.476888 30008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb66fb2351f442c4bd3137274cdb5449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.477102 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb66fb2351f442c4bd3137274cdb5449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.484249 30138 tablet_service.cc:1467] Processing CreateTablet for tablet cb66fb2351f442c4bd3137274cdb5449 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6336a199bdbc48389a3a983413491684]), partition=
I20240502 08:18:03.484598 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb66fb2351f442c4bd3137274cdb5449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.489352 32066 ts_tablet_manager.cc:1888] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:03.489480 32066 log.cc:1196] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/a7fc48f77b044ce8bc9e74ecc0ef1d83
I20240502 08:18:03.489931 32066 ts_tablet_manager.cc:1909] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting consensus metadata
I20240502 08:18:03.493394 31962 tablet_bootstrap.cc:492] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:03.495281 31962 tablet_bootstrap.cc:654] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.495934 27519 raft_consensus.cc:1203] T e67c99e57a73443da13527ed33326a4e P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:03.496261 32067 ts_tablet_manager.cc:1888] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:03.497287 32067 log.cc:1196] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/a7fc48f77b044ce8bc9e74ecc0ef1d83
I20240502 08:18:03.497735 32067 ts_tablet_manager.cc:1909] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P 1f0bf4bc3c5146539146b825b048bff3: Deleting consensus metadata
I20240502 08:18:03.498975 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet a7fc48f77b044ce8bc9e74ecc0ef1d83 (table rebalancer_dynamic_table_new_name [id=0223e428283e4982928c2c5a2e70edf5]) successfully deleted
I20240502 08:18:03.499876 31963 tablet_bootstrap.cc:492] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:03.501192 31963 tablet_bootstrap.cc:654] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.503911 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet a7fc48f77b044ce8bc9e74ecc0ef1d83 (table rebalancer_dynamic_table_new_name [id=0223e428283e4982928c2c5a2e70edf5]) successfully deleted
I20240502 08:18:03.505190 31963 tablet_bootstrap.cc:492] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:03.505324 31963 ts_tablet_manager.cc:1367] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:18:03.506146 31963 raft_consensus.cc:348] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:03.506376 31963 raft_consensus.cc:374] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:03.506491 31963 raft_consensus.cc:729] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:03.507073 31963 consensus_queue.cc:260] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:03.509331 31963 ts_tablet_manager.cc:1398] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:03.510952 32007 tablet_bootstrap.cc:492] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:03.511416 32042 tablet_bootstrap.cc:492] T 201d79895c2741dd9d9de0cc9f605a91 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:03.512864 32042 tablet_bootstrap.cc:492] T 201d79895c2741dd9d9de0cc9f605a91 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap complete.
I20240502 08:18:03.513092 32068 ts_tablet_manager.cc:1888] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:03.513659 32068 log.cc:1196] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/a7fc48f77b044ce8bc9e74ecc0ef1d83
I20240502 08:18:03.515128 32068 ts_tablet_manager.cc:1909] T a7fc48f77b044ce8bc9e74ecc0ef1d83 P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:18:03.516733 32079 tablet_bootstrap.cc:492] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:03.520092 32079 tablet_bootstrap.cc:654] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.522106 32042 ts_tablet_manager.cc:1367] T 201d79895c2741dd9d9de0cc9f605a91 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.005s
I20240502 08:18:03.524747 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet a7fc48f77b044ce8bc9e74ecc0ef1d83 (table rebalancer_dynamic_table_new_name [id=0223e428283e4982928c2c5a2e70edf5]) successfully deleted
I20240502 08:18:03.527222 32042 raft_consensus.cc:348] T 201d79895c2741dd9d9de0cc9f605a91 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:03.527488 32042 raft_consensus.cc:729] T 201d79895c2741dd9d9de0cc9f605a91 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: LEARNER
I20240502 08:18:03.528121 32007 tablet_bootstrap.cc:492] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap complete.
I20240502 08:18:03.528295 31962 tablet_bootstrap.cc:492] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:03.528404 32007 ts_tablet_manager.cc:1367] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.277s user 0.050s sys 0.003s
I20240502 08:18:03.528604 31962 ts_tablet_manager.cc:1367] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.005s
I20240502 08:18:03.528877 32042 consensus_queue.cc:260] T 201d79895c2741dd9d9de0cc9f605a91 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:03.529474 32042 ts_tablet_manager.cc:1398] T 201d79895c2741dd9d9de0cc9f605a91 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 08:18:03.530251 32079 tablet_bootstrap.cc:492] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:03.530427 32079 ts_tablet_manager.cc:1367] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20240502 08:18:03.531075 30443 tablet_copy_service.cc:342] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Request end of tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-201d79895c2741dd9d9de0cc9f605a91 received from {username='slave'} at 127.20.207.2:33397
I20240502 08:18:03.531204 30443 tablet_copy_service.cc:433] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: ending tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-201d79895c2741dd9d9de0cc9f605a91 on tablet 201d79895c2741dd9d9de0cc9f605a91 with peer 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:03.531195 32007 raft_consensus.cc:348] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:03.531477 32007 raft_consensus.cc:729] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: LEARNER
I20240502 08:18:03.531772 32079 raft_consensus.cc:348] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:03.531679 32007 consensus_queue.cc:260] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:03.532325 32007 ts_tablet_manager.cc:1398] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:03.533600 32079 raft_consensus.cc:374] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:03.533751 32079 raft_consensus.cc:729] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:03.534050 32079 consensus_queue.cc:260] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:03.535560 32079 ts_tablet_manager.cc:1398] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:03.536697 27671 tablet_copy_service.cc:342] P bf8ac71b15f740a0a928911360773442: Request end of tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-cad036d297bd45d7b7c5a1f19356906f received from {username='slave'} at 127.20.207.8:38973
I20240502 08:18:03.536809 27671 tablet_copy_service.cc:433] P bf8ac71b15f740a0a928911360773442: ending tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-cad036d297bd45d7b7c5a1f19356906f on tablet cad036d297bd45d7b7c5a1f19356906f with peer 31a1e33dcdbf42f58ea72ce1c0afb0ad
I20240502 08:18:03.547679 32008 tablet_bootstrap.cc:492] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:03.548043 32008 tablet_bootstrap.cc:492] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap complete.
I20240502 08:18:03.548281 32008 ts_tablet_manager.cc:1367] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.194s user 0.047s sys 0.000s
I20240502 08:18:03.548847 32008 raft_consensus.cc:348] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:03.549070 32008 raft_consensus.cc:729] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: LEARNER
I20240502 08:18:03.549283 32008 consensus_queue.cc:260] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:03.529585 31962 raft_consensus.cc:348] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:03.549726 31962 raft_consensus.cc:374] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:03.549809 31962 raft_consensus.cc:729] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:03.549820 32008 ts_tablet_manager.cc:1398] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 08:18:03.550060 31962 consensus_queue.cc:260] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:03.550491 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 1f0bf4bc3c5146539146b825b048bff3-0bea60d096f9408c9f4a857f8824d6a2 received from {username='slave'} at 127.20.207.5:54501
I20240502 08:18:03.550571 31962 ts_tablet_manager.cc:1398] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.022s user 0.001s sys 0.000s
I20240502 08:18:03.550609 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 1f0bf4bc3c5146539146b825b048bff3-0bea60d096f9408c9f4a857f8824d6a2 on tablet 0bea60d096f9408c9f4a857f8824d6a2 with peer 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:18:03.554513 31947 tablet_bootstrap.cc:492] T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:03.554980 31947 tablet_bootstrap.cc:492] T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap complete.
I20240502 08:18:03.555279 31947 ts_tablet_manager.cc:1367] T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.387s user 0.043s sys 0.007s
I20240502 08:18:03.555711 30290 raft_consensus.cc:1203] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:03.556128 31947 raft_consensus.cc:348] T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:03.556486 31947 raft_consensus.cc:729] T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: LEARNER
I20240502 08:18:03.556804 31947 consensus_queue.cc:260] T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:03.558897 31947 ts_tablet_manager.cc:1398] T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:03.560101 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-6978aa3955504c548778066ddd0910e5 received from {username='slave'} at 127.20.207.8:40413
I20240502 08:18:03.560276 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-6978aa3955504c548778066ddd0910e5 on tablet 6978aa3955504c548778066ddd0910e5 with peer 31a1e33dcdbf42f58ea72ce1c0afb0ad
I20240502 08:18:03.562922 30423 raft_consensus.cc:1203] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:03.592869 27390 raft_consensus.cc:1203] T 201d79895c2741dd9d9de0cc9f605a91 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:03.632988 27258 raft_consensus.cc:1203] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:03.640514 31777 raft_consensus.cc:482] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:03.640719 31777 raft_consensus.cc:504] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:03.641592 31777 leader_election.cc:288] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:03.642333 30156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb66fb2351f442c4bd3137274cdb5449" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:03.642576 30156 raft_consensus.cc:2454] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:03.642503 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb66fb2351f442c4bd3137274cdb5449" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:03.642784 30290 raft_consensus.cc:2454] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:03.643157 29964 leader_election.cc:302] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:03.643688 31777 raft_consensus.cc:2781] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:03.643839 31777 raft_consensus.cc:482] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:03.643963 31777 raft_consensus.cc:3037] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.648226 31777 raft_consensus.cc:504] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:03.649051 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb66fb2351f442c4bd3137274cdb5449" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:03.649268 30290 raft_consensus.cc:3037] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.649371 30156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb66fb2351f442c4bd3137274cdb5449" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:03.649555 30156 raft_consensus.cc:3037] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.652048 30290 raft_consensus.cc:2454] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:03.652230 30156 raft_consensus.cc:2454] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:03.652645 29964 leader_election.cc:302] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:03.652921 32043 raft_consensus.cc:2781] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:03.653061 32043 raft_consensus.cc:686] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:18:03.653322 32043 consensus_queue.cc:237] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:03.655064 31777 leader_election.cc:288] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:03.667399 27110 catalog_manager.cc:5484] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:03.716194 31920 consensus_queue.cc:1035] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:03.723266 31920 consensus_queue.cc:1035] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:03.747427 30026 raft_consensus.cc:1203] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:03.758960 30423 raft_consensus.cc:1203] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:03.768764 31864 raft_consensus.cc:1050] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: attempting to promote NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 to VOTER
I20240502 08:18:03.769698 31864 consensus_queue.cc:237] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:03.773905 27258 raft_consensus.cc:1261] T 83993ff2073842988937597dd3999d0d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:03.774477 31872 consensus_queue.cc:1035] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:03.777015 30156 raft_consensus.cc:1261] T 83993ff2073842988937597dd3999d0d P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:03.777626 31864 consensus_queue.cc:1035] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:03.777740 30026 raft_consensus.cc:1261] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:03.783474 31864 consensus_queue.cc:1035] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:03.790823 31872 raft_consensus.cc:2932] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:03.791492 30026 raft_consensus.cc:2932] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:03.796473 27258 raft_consensus.cc:2932] T 83993ff2073842988937597dd3999d0d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:03.798703 27110 catalog_manager.cc:5484] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:03.806099 27109 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:03.807530 30156 raft_consensus.cc:2932] T 83993ff2073842988937597dd3999d0d P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:03.810724 31689 raft_consensus.cc:1050] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER 84cda20738d2452da166e31fc6da57a6 to VOTER
I20240502 08:18:03.811520 31689 consensus_queue.cc:237] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:03.812992 31540 raft_consensus.cc:1050] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 to VOTER
I20240502 08:18:03.813825 31540 consensus_queue.cc:237] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:03.814788 30423 consensus_queue.cc:237] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:03.817018 27257 raft_consensus.cc:1261] T 83993ff2073842988937597dd3999d0d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:03.817735 30158 raft_consensus.cc:1261] T 83993ff2073842988937597dd3999d0d P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:03.818801 31872 consensus_queue.cc:1035] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:03.819821 31872 consensus_queue.cc:1035] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:03.821404 27258 consensus_queue.cc:237] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:03.831121 32162 raft_consensus.cc:2932] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
W20240502 08:18:03.832227 27194 consensus_peers.cc:485] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Couldn't send request to peer 84cda20738d2452da166e31fc6da57a6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a793eaaeba89458baf8af7caf3f24c70. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:03.833158 27390 raft_consensus.cc:1261] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:03.833264 31666 consensus_queue.cc:1035] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:03.834188 31680 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:03.834908 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 83993ff2073842988937597dd3999d0d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:03.835258 27257 raft_consensus.cc:2932] T 83993ff2073842988937597dd3999d0d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:03.837363 30158 raft_consensus.cc:2932] T 83993ff2073842988937597dd3999d0d P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:03.839309 27109 catalog_manager.cc:5484] T 83993ff2073842988937597dd3999d0d P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:03.845016 31920 consensus_queue.cc:1035] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:03.845165 27519 raft_consensus.cc:1261] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:03.845860 31680 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:03.846630 30007 tablet_service.cc:1514] Processing DeleteTablet for tablet 83993ff2073842988937597dd3999d0d with delete_type TABLET_DATA_TOMBSTONED (TS 1f0bf4bc3c5146539146b825b048bff3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:03.847033 32067 tablet_replica.cc:330] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:03.847328 32067 raft_consensus.cc:2229] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:03.847612 32067 raft_consensus.cc:2258] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:03.848623 32067 ts_tablet_manager.cc:1875] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:03.862442 32067 ts_tablet_manager.cc:1888] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:03.862685 32067 log.cc:1196] T 83993ff2073842988937597dd3999d0d P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/83993ff2073842988937597dd3999d0d
I20240502 08:18:03.863565 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet 83993ff2073842988937597dd3999d0d (table rebalancer_extra_table_6 [id=e57c96062aca4609bc2d923cb7a30a29]) successfully deleted
I20240502 08:18:03.865388 31680 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:03.865911 27519 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:03.873935 30158 raft_consensus.cc:1261] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:03.874635 32043 consensus_queue.cc:1035] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:03.882350 31920 consensus_queue.cc:1035] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:03.883783 30288 raft_consensus.cc:1261] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:03.884397 32043 consensus_queue.cc:1035] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:03.887718 27110 catalog_manager.cc:5484] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:03.898065 30026 raft_consensus.cc:1261] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:03.900156 31682 consensus_queue.cc:1035] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:03.904340 27193 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.2:46291, user_credentials={real_user=slave}} blocked reactor thread for 46896us
I20240502 08:18:03.906764 27390 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:03.919555 30290 raft_consensus.cc:1203] T a6dce110666a4a038f5f46d48a3ec7af P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:03.906067 32171 consensus_queue.cc:1035] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20240502 08:18:03.923689 27108 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet cb66fb2351f442c4bd3137274cdb5449 (table rebalancer_dynamic_table [id=6336a199bdbc48389a3a983413491684]). Expected version 1 got 0
I20240502 08:18:03.925144 32186 tablet.cc:1722] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:03.925891 32186 tablet.cc:1620] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:03.927480 27108 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:03.936829 27109 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=6336a199bdbc48389a3a983413491684] alter complete (version 1)
I20240502 08:18:03.938611 32185 tablet.cc:1722] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:03.939366 27111 catalog_manager.cc:5576] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899) does not have the latest schema for tablet cb66fb2351f442c4bd3137274cdb5449 (table rebalancer_dynamic_table [id=6336a199bdbc48389a3a983413491684]). Expected version 1 got 0
I20240502 08:18:03.940238 27370 tablet_service.cc:1467] Processing CreateTablet for tablet 8d309785afd8467dab3418bb63c95030 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=77ecd1036c454af293decc2ea5173f97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:03.940680 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d309785afd8467dab3418bb63c95030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.941278 30400 tablet_service.cc:1467] Processing CreateTablet for tablet 8d309785afd8467dab3418bb63c95030 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=77ecd1036c454af293decc2ea5173f97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:03.941649 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d309785afd8467dab3418bb63c95030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.942247 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 039b1aea12ba4627882b603336070773 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=77ecd1036c454af293decc2ea5173f97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:03.942678 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039b1aea12ba4627882b603336070773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.946424 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 039b1aea12ba4627882b603336070773 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=77ecd1036c454af293decc2ea5173f97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:03.946792 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039b1aea12ba4627882b603336070773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.948087 32187 tablet_bootstrap.cc:492] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:03.941215 30008 tablet_service.cc:1467] Processing CreateTablet for tablet 039b1aea12ba4627882b603336070773 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=77ecd1036c454af293decc2ea5173f97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:03.950238 30008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039b1aea12ba4627882b603336070773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.950315 32187 tablet_bootstrap.cc:654] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.956265 30007 tablet_service.cc:1467] Processing CreateTablet for tablet 8d309785afd8467dab3418bb63c95030 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=77ecd1036c454af293decc2ea5173f97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:03.956282 30270 tablet_service.cc:1467] Processing CreateTablet for tablet 65ac899c820a43e4ab57454e224edc83 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=77ecd1036c454af293decc2ea5173f97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:03.956372 32082 raft_consensus.cc:1050] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442: attempting to promote NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad to VOTER
I20240502 08:18:03.956629 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65ac899c820a43e4ab57454e224edc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.956656 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d309785afd8467dab3418bb63c95030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.957124 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 65ac899c820a43e4ab57454e224edc83 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=77ecd1036c454af293decc2ea5173f97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:03.957192 32082 consensus_queue.cc:237] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:03.957502 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65ac899c820a43e4ab57454e224edc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.958350 30006 tablet_service.cc:1467] Processing CreateTablet for tablet 65ac899c820a43e4ab57454e224edc83 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=77ecd1036c454af293decc2ea5173f97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:03.958710 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65ac899c820a43e4ab57454e224edc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.962741 31962 tablet_bootstrap.cc:492] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:03.964449 31962 tablet_bootstrap.cc:654] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.966658 32079 tablet_bootstrap.cc:492] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:03.968199 32079 tablet_bootstrap.cc:654] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.969415 27110 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet cb66fb2351f442c4bd3137274cdb5449 (table rebalancer_dynamic_table [id=6336a199bdbc48389a3a983413491684]). Expected version 1 got 0
W20240502 08:18:03.971892 27191 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.5:37815, user_credentials={real_user=slave}} blocked reactor thread for 73243.7us
I20240502 08:18:03.979251 27110 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet cb66fb2351f442c4bd3137274cdb5449 (table rebalancer_dynamic_table [id=6336a199bdbc48389a3a983413491684]). Expected version 1 got 0
I20240502 08:18:03.982474 32185 tablet.cc:1620] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:03.982586 32188 tablet_bootstrap.cc:492] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:03.982995 32194 tablet.cc:1722] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240502 08:18:03.983196 27194 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.3:44469, user_credentials={real_user=slave}} blocked reactor thread for 99915us
I20240502 08:18:03.985720 32192 tablet_bootstrap.cc:492] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:03.985750 27110 catalog_manager.cc:5576] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899) does not have the latest schema for tablet cb66fb2351f442c4bd3137274cdb5449 (table rebalancer_dynamic_table [id=6336a199bdbc48389a3a983413491684]). Expected version 1 got 0
I20240502 08:18:03.988232 27519 raft_consensus.cc:1261] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:03.988247 27648 raft_consensus.cc:1261] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:03.989013 32188 tablet_bootstrap.cc:654] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.990137 27648 raft_consensus.cc:1261] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:03.990177 27519 raft_consensus.cc:1261] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:03.995648 32192 tablet_bootstrap.cc:654] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.997092 32172 consensus_queue.cc:1035] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:03.997447 32194 tablet.cc:1620] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:18:03.997674 32172 consensus_queue.cc:1035] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:03.999584 32172 consensus_queue.cc:1035] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.000646 32172 consensus_queue.cc:1035] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:03.992707 30158 raft_consensus.cc:1261] T af2f11471c064f48924191fc2607d31c P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.004973 32182 raft_consensus.cc:1050] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER b01e7c8a302d41078fa39cfef665ff59 to VOTER
I20240502 08:18:04.006088 32182 consensus_queue.cc:237] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:04.007879 30423 raft_consensus.cc:1203] T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:04.013839 32188 tablet_bootstrap.cc:492] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:04.014227 32188 ts_tablet_manager.cc:1367] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.001s
I20240502 08:18:04.015302 32188 raft_consensus.cc:348] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.015728 32188 raft_consensus.cc:374] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.015887 32188 raft_consensus.cc:729] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.016227 32188 consensus_queue.cc:260] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.016855 30423 raft_consensus.cc:1261] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.017148 27519 raft_consensus.cc:1261] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.017763 31753 consensus_queue.cc:1035] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.018406 27389 raft_consensus.cc:1261] T cad036d297bd45d7b7c5a1f19356906f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.018738 32081 consensus_queue.cc:1035] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.019418 32082 consensus_queue.cc:1035] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.023334 31864 raft_consensus.cc:1050] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: attempting to promote NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 to VOTER
I20240502 08:18:04.023824 31705 raft_consensus.cc:1050] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 to VOTER
I20240502 08:18:04.024101 31864 consensus_queue.cc:237] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:04.024271 32192 tablet_bootstrap.cc:492] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:18:04.024405 32192 ts_tablet_manager.cc:1367] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20240502 08:18:04.024590 31705 consensus_queue.cc:237] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:04.025219 32192 raft_consensus.cc:348] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.025432 32192 raft_consensus.cc:374] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.025534 32192 raft_consensus.cc:729] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.025827 32192 consensus_queue.cc:260] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.027251 32192 ts_tablet_manager.cc:1398] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:18:04.027411 32188 ts_tablet_manager.cc:1398] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240502 08:18:04.037504 32187 tablet_bootstrap.cc:492] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:04.037711 32187 ts_tablet_manager.cc:1367] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.090s user 0.000s sys 0.007s
I20240502 08:18:04.039340 32187 raft_consensus.cc:348] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:04.039661 32187 raft_consensus.cc:374] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.039819 32187 raft_consensus.cc:729] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.040194 32187 consensus_queue.cc:260] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:04.042574 27108 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:04.043785 27387 raft_consensus.cc:1261] T 201d79895c2741dd9d9de0cc9f605a91 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:04.044450 31309 consensus_queue.cc:1035] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:04.045645 32187 ts_tablet_manager.cc:1398] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 08:18:04.045962 32187 tablet_bootstrap.cc:492] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:04.048995 32187 tablet_bootstrap.cc:654] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.051231 30156 raft_consensus.cc:1261] T 201d79895c2741dd9d9de0cc9f605a91 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:04.051826 31309 consensus_queue.cc:1035] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:04.056084 27258 raft_consensus.cc:1261] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.056712 31359 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.058696 27648 raft_consensus.cc:1261] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:04.059221 31309 consensus_queue.cc:1035] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:04.061272 32185 tablet.cc:1722] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:04.061900 32185 tablet.cc:1620] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:04.063131 30290 raft_consensus.cc:1261] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.063688 32194 tablet.cc:1722] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:04.063771 31359 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.064266 32194 tablet.cc:1620] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:18:04.065250 27390 raft_consensus.cc:1261] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.066103 31719 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.069715 31848 raft_consensus.cc:482] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:04.069876 31848 raft_consensus.cc:504] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.070708 31848 leader_election.cc:288] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:04.071601 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d309785afd8467dab3418bb63c95030" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:04.071856 30421 raft_consensus.cc:2454] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:18:04.072295 27323 leader_election.cc:302] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:04.072981 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d309785afd8467dab3418bb63c95030" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
W20240502 08:18:04.077062 27323 leader_election.cc:341] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:04.077504 31359 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:04.078332 27111 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=6336a199bdbc48389a3a983413491684] alter complete (version 2)
I20240502 08:18:04.079392 31848 raft_consensus.cc:2781] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:04.079536 31848 raft_consensus.cc:482] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:04.079645 31848 raft_consensus.cc:3037] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.082337 32187 tablet_bootstrap.cc:492] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:04.082578 31912 raft_consensus.cc:1050] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3: attempting to promote NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 to VOTER
I20240502 08:18:04.082504 32187 ts_tablet_manager.cc:1367] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.000s
I20240502 08:18:04.083109 27388 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:04.083565 32187 raft_consensus.cc:348] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.083525 31912 consensus_queue.cc:237] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:04.083766 32187 raft_consensus.cc:374] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.083843 32187 raft_consensus.cc:729] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.084086 32187 consensus_queue.cc:260] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.084692 32187 ts_tablet_manager.cc:1398] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:04.087860 27257 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:04.086226 31848 raft_consensus.cc:504] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.091226 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d309785afd8467dab3418bb63c95030" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:04.091460 30421 raft_consensus.cc:3037] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.093385 27111 catalog_manager.cc:5484] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:04.096020 32079 tablet_bootstrap.cc:492] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:04.096187 32079 ts_tablet_manager.cc:1367] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.130s user 0.008s sys 0.004s
I20240502 08:18:04.096254 32186 tablet.cc:1722] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:04.096711 32186 tablet.cc:1620] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:04.096717 32079 raft_consensus.cc:348] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.096908 32079 raft_consensus.cc:374] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.097010 32079 raft_consensus.cc:729] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.097242 32079 consensus_queue.cc:260] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.097754 30421 raft_consensus.cc:2454] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:18:04.098244 27323 leader_election.cc:302] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:04.098718 32152 raft_consensus.cc:2781] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:04.105460 31753 raft_consensus.cc:2932] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:04.110472 27518 raft_consensus.cc:2932] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:04.110791 32162 raft_consensus.cc:2932] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:04.116669 32175 consensus_queue.cc:1035] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.117975 30156 raft_consensus.cc:2932] T 201d79895c2741dd9d9de0cc9f605a91 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:04.126505 27111 catalog_manager.cc:5484] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:04.130550 31848 leader_election.cc:288] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:04.138263 27389 raft_consensus.cc:2932] T 201d79895c2741dd9d9de0cc9f605a91 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:04.140864 27652 raft_consensus.cc:2932] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:04.142086 30421 consensus_queue.cc:237] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:04.144438 30159 raft_consensus.cc:1261] T 201d79895c2741dd9d9de0cc9f605a91 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:04.145044 31864 consensus_queue.cc:1035] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:04.148808 31864 raft_consensus.cc:2932] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:04.153725 31962 tablet_bootstrap.cc:492] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
W20240502 08:18:04.154698 32202 log.cc:924] Time spent T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20240502 08:18:04.156944 30423 raft_consensus.cc:2932] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:04.157177 27387 raft_consensus.cc:2932] T cad036d297bd45d7b7c5a1f19356906f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:04.158107 31753 raft_consensus.cc:482] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:04.158387 31753 raft_consensus.cc:504] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:04.159771 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039b1aea12ba4627882b603336070773" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:04.160076 32081 raft_consensus.cc:482] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:04.160259 32081 raft_consensus.cc:504] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.160542 31962 ts_tablet_manager.cc:1367] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.198s user 0.008s sys 0.006s
I20240502 08:18:04.160936 32081 leader_election.cc:288] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
W20240502 08:18:04.161515 27584 leader_election.cc:341] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:04.161574 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ac899c820a43e4ab57454e224edc83" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
W20240502 08:18:04.162287 27584 leader_election.cc:341] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:04.162746 31962 raft_consensus.cc:348] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.162997 31962 raft_consensus.cc:374] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.163182 31962 raft_consensus.cc:729] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.161980 31753 leader_election.cc:288] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:04.169988 27109 catalog_manager.cc:5484] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
W20240502 08:18:04.179651 27323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.7:40343, user_credentials={real_user=slave}} blocked reactor thread for 81110.7us
I20240502 08:18:04.180500 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 201d79895c2741dd9d9de0cc9f605a91 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:04.185801 31680 raft_consensus.cc:2932] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:04.186604 27388 raft_consensus.cc:1261] T 201d79895c2741dd9d9de0cc9f605a91 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:04.190308 30290 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:04.190666 27109 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:04.192853 31848 raft_consensus.cc:1050] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1: attempting to promote NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad to VOTER
I20240502 08:18:04.193454 27110 catalog_manager.cc:5484] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 3 to 4, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:04.193634 31848 consensus_queue.cc:237] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:04.194873 30158 raft_consensus.cc:2932] T 201d79895c2741dd9d9de0cc9f605a91 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:04.198299 30289 raft_consensus.cc:1261] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:04.199174 32175 consensus_queue.cc:1035] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:04.200584 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet 201d79895c2741dd9d9de0cc9f605a91 with delete_type TABLET_DATA_TOMBSTONED (TS bf8ac71b15f740a0a928911360773442 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44788
I20240502 08:18:04.204106 27109 catalog_manager.cc:5484] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:04.209808 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039b1aea12ba4627882b603336070773" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:04.210322 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ac899c820a43e4ab57454e224edc83" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
W20240502 08:18:04.213739 27584 leader_election.cc:341] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:04.213879 27584 leader_election.cc:302] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 1f0bf4bc3c5146539146b825b048bff3, 49c5c8f70e494edd9fcda40cc0af6cc4
W20240502 08:18:04.214397 27584 leader_election.cc:341] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:04.214520 27584 leader_election.cc:302] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 1f0bf4bc3c5146539146b825b048bff3, 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:04.215312 32081 raft_consensus.cc:2726] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:04.215366 31753 raft_consensus.cc:2726] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:04.216066 32079 ts_tablet_manager.cc:1398] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.120s user 0.000s sys 0.002s
I20240502 08:18:04.216912 31680 raft_consensus.cc:2932] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:04.217581 30025 raft_consensus.cc:2932] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:04.222200 31962 consensus_queue.cc:260] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.224156 31962 ts_tablet_manager.cc:1398] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.063s user 0.000s sys 0.002s
I20240502 08:18:04.234939 31309 consensus_queue.cc:1035] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:18:04.237612 27194 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.3:44469, user_credentials={real_user=slave}} blocked reactor thread for 51265us
I20240502 08:18:04.237582 27108 catalog_manager.cc:5484] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:04.240017 30156 raft_consensus.cc:2932] T af2f11471c064f48924191fc2607d31c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:04.240612 30159 raft_consensus.cc:1261] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:04.240923 30026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d309785afd8467dab3418bb63c95030" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:04.241189 30026 raft_consensus.cc:3037] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.242949 27257 consensus_queue.cc:237] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:04.246697 30025 raft_consensus.cc:1261] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:04.247285 32179 consensus_queue.cc:1035] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:04.248441 32152 raft_consensus.cc:686] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:18:04.248880 32152 consensus_queue.cc:237] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.249202 27519 raft_consensus.cc:1261] T e67c99e57a73443da13527ed33326a4e P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:04.250646 32227 tablet_replica.cc:330] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:18:04.250500 32175 raft_consensus.cc:2932] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:04.250985 32227 raft_consensus.cc:2229] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.251298 32227 raft_consensus.cc:2258] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.252406 32227 ts_tablet_manager.cc:1875] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:04.253743 31962 tablet_bootstrap.cc:492] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:04.254550 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dd887cbb35c74e54a43d3dc6a2dde817 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:04.255702 30158 raft_consensus.cc:1261] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:04.256397 32043 consensus_queue.cc:1035] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:04.256083 30025 raft_consensus.cc:2932] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:04.261937 32079 tablet_bootstrap.cc:492] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:04.264110 27389 raft_consensus.cc:2932] T 201d79895c2741dd9d9de0cc9f605a91 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:04.266330 27520 consensus_queue.cc:237] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:04.268299 31962 tablet_bootstrap.cc:654] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.269595 27518 raft_consensus.cc:2932] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:04.270203 27519 raft_consensus.cc:2932] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:04.271142 30288 raft_consensus.cc:1261] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:04.273325 32043 consensus_queue.cc:1035] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:04.275986 31719 raft_consensus.cc:1050] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 to VOTER
I20240502 08:18:04.276798 31719 consensus_queue.cc:237] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:04.280388 27652 consensus_queue.cc:237] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:04.282989 27257 consensus_queue.cc:237] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:04.283336 30423 raft_consensus.cc:1261] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:04.283644 27108 catalog_manager.cc:5484] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:04.284126 31753 consensus_queue.cc:1035] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:04.285544 30026 raft_consensus.cc:2454] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:18:04.288347 27258 raft_consensus.cc:1261] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.289315 27255 raft_consensus.cc:1261] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:04.289239 31705 consensus_queue.cc:1035] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.290746 27108 catalog_manager.cc:5484] T dd887cbb35c74e54a43d3dc6a2dde817 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:04.293735 32081 raft_consensus.cc:2932] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:04.294209 32185 tablet.cc:1722] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:04.299753 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cad036d297bd45d7b7c5a1f19356906f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:04.300851 31962 tablet_bootstrap.cc:492] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:04.301048 31962 ts_tablet_manager.cc:1367] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.001s
I20240502 08:18:04.300958 32194 tablet.cc:1722] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:04.302410 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet dd887cbb35c74e54a43d3dc6a2dde817 with delete_type TABLET_DATA_TOMBSTONED (TS bf8ac71b15f740a0a928911360773442 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44788
I20240502 08:18:04.304044 30423 raft_consensus.cc:2932] T cad036d297bd45d7b7c5a1f19356906f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:04.308750 31777 consensus_queue.cc:1035] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:04.310686 31962 raft_consensus.cc:348] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:04.310925 31962 raft_consensus.cc:374] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.311007 31962 raft_consensus.cc:729] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.311503 27388 raft_consensus.cc:1261] T cad036d297bd45d7b7c5a1f19356906f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:04.312538 32082 consensus_queue.cc:1035] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:04.316852 31962 consensus_queue.cc:260] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:04.317369 27111 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=6336a199bdbc48389a3a983413491684] alter complete (version 3)
I20240502 08:18:04.317456 30026 raft_consensus.cc:1261] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.322021 27388 raft_consensus.cc:2932] T cad036d297bd45d7b7c5a1f19356906f P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:04.322381 31962 ts_tablet_manager.cc:1398] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
I20240502 08:18:04.322521 27108 catalog_manager.cc:5484] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:04.329239 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet cad036d297bd45d7b7c5a1f19356906f with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:04.330852 27648 raft_consensus.cc:2932] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:04.332417 32214 consensus_queue.cc:1035] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.343246 31777 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
W20240502 08:18:04.344969 27194 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.6:38899, user_credentials={real_user=slave}} blocked reactor thread for 74977.4us
I20240502 08:18:04.345295 32175 consensus_queue.cc:1035] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:04.346622 32242 tablet_replica.cc:330] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:04.346920 32242 raft_consensus.cc:2229] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.347416 32242 raft_consensus.cc:2258] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.348469 32242 ts_tablet_manager.cc:1875] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:04.351229 27109 catalog_manager.cc:5484] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:04.351903 27258 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:04.352840 30288 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:04.356779 27521 raft_consensus.cc:1261] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.356782 30421 raft_consensus.cc:1261] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.359090 32216 consensus_queue.cc:1035] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.360101 32152 consensus_queue.cc:1035] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.360590 27257 raft_consensus.cc:1261] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:04.360824 30156 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:04.361387 31719 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:04.364188 27521 raft_consensus.cc:1261] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:04.364004 27257 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:04.364739 32186 tablet.cc:1722] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:04.365456 30288 raft_consensus.cc:1261] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:04.365847 32182 consensus_queue.cc:1035] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:04.366369 32182 consensus_queue.cc:1035] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:04.367313 30158 raft_consensus.cc:1261] T af2f11471c064f48924191fc2607d31c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:04.370244 32079 tablet_bootstrap.cc:654] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.380128 31705 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:04.383905 27517 consensus_queue.cc:237] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:04.390666 27112 catalog_manager.cc:5484] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 5 to 6, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:04.396015 32175 consensus_queue.cc:1035] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240502 08:18:04.399755 32227 ts_tablet_manager.cc:1888] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:04.399976 32227 log.cc:1196] T 201d79895c2741dd9d9de0cc9f605a91 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/201d79895c2741dd9d9de0cc9f605a91
I20240502 08:18:04.400077 31359 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:04.401669 30288 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:04.401474 30025 consensus_queue.cc:237] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:04.402935 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 201d79895c2741dd9d9de0cc9f605a91 (table rebalancer_extra_table_8 [id=05e7c365fc854833a30f18097c117d10]) successfully deleted
I20240502 08:18:04.404103 32227 tablet_replica.cc:330] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:18:04.404946 30158 raft_consensus.cc:1261] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:04.405656 31912 consensus_queue.cc:1035] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:04.408529 30160 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:04.409096 32230 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:04.414438 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 26545a868bd6415c9c9db4869b2841c7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:04.416276 32227 raft_consensus.cc:2229] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.416262 27110 catalog_manager.cc:5484] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:04.421281 32227 pending_rounds.cc:62] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442: Trying to abort 1 pending ops.
I20240502 08:18:04.421391 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet 26545a868bd6415c9c9db4869b2841c7 with delete_type TABLET_DATA_TOMBSTONED (TS 49c5c8f70e494edd9fcda40cc0af6cc4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:04.421715 27518 raft_consensus.cc:1261] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:04.421468 32227 pending_rounds.cc:68] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023156772099272704 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "dd887cbb35c74e54a43d3dc6a2dde817" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } } }
I20240502 08:18:04.422689 32227 raft_consensus.cc:2894] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 08:18:04.422873 32227 raft_consensus.cc:2866] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 08:18:04.423106 32227 raft_consensus.cc:2258] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.424870 32227 ts_tablet_manager.cc:1875] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:04.425343 32079 tablet_bootstrap.cc:492] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:04.428066 27109 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:04.428316 27109 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:04.430999 31680 raft_consensus.cc:2932] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:04.436211 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af2f11471c064f48924191fc2607d31c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:04.436897 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 09aa1b5d99114b3095be08c9547fbb54 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:04.440852 27648 raft_consensus.cc:1261] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:04.441440 32214 consensus_queue.cc:1035] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:04.444744 32254 tablet_replica.cc:330] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:04.445031 32254 raft_consensus.cc:2229] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.446035 32079 ts_tablet_manager.cc:1367] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.184s user 0.000s sys 0.004s
I20240502 08:18:04.446460 32244 ts_tablet_manager.cc:902] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:04.443990 27110 catalog_manager.cc:5484] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:04.448494 27517 raft_consensus.cc:2932] T af2f11471c064f48924191fc2607d31c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:04.450989 31680 consensus_queue.cc:1035] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:04.451148 27109 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet cb66fb2351f442c4bd3137274cdb5449
I20240502 08:18:04.451881 30006 tablet_service.cc:1514] Processing DeleteTablet for tablet cb66fb2351f442c4bd3137274cdb5449 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:04.445696 32254 raft_consensus.cc:2258] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.452210 30138 tablet_service.cc:1514] Processing DeleteTablet for tablet cb66fb2351f442c4bd3137274cdb5449 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:04.452129 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet cb66fb2351f442c4bd3137274cdb5449 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:52730
W20240502 08:18:04.452368 27453 consensus_peers.cc:485] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 -> Peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Couldn't send request to peer 1f0bf4bc3c5146539146b825b048bff3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e0313e85d444e098bfa35be89a3ad97. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:04.448441 32246 raft_consensus.cc:2932] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:04.452754 30026 raft_consensus.cc:2932] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
W20240502 08:18:04.458596 27455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.1:38505, user_credentials={real_user=slave}} blocked reactor thread for 84278.9us
I20240502 08:18:04.459735 27257 raft_consensus.cc:1261] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:04.460079 32254 ts_tablet_manager.cc:1875] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:04.460997 32230 consensus_queue.cc:1035] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:04.462208 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet af2f11471c064f48924191fc2607d31c with delete_type TABLET_DATA_TOMBSTONED (TS bf8ac71b15f740a0a928911360773442 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44788
I20240502 08:18:04.463304 27258 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:04.466802 32079 raft_consensus.cc:348] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:04.467077 27109 catalog_manager.cc:5484] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:04.467864 32175 raft_consensus.cc:2932] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:04.473342 30288 raft_consensus.cc:2932] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:04.482004 30157 raft_consensus.cc:2932] T af2f11471c064f48924191fc2607d31c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:04.482823 27521 raft_consensus.cc:2932] T dd887cbb35c74e54a43d3dc6a2dde817 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:04.483286 27111 catalog_manager.cc:5484] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:04.490633 27520 raft_consensus.cc:2932] T e67c99e57a73443da13527ed33326a4e P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:04.478478 32257 tablet_replica.cc:330] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:04.496271 32257 raft_consensus.cc:2229] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:04.496711 32257 raft_consensus.cc:2258] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.497589 32242 ts_tablet_manager.cc:1888] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:04.497761 32257 ts_tablet_manager.cc:1875] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.502092 32242 log.cc:1196] T cad036d297bd45d7b7c5a1f19356906f P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/cad036d297bd45d7b7c5a1f19356906f
I20240502 08:18:04.502445 32079 raft_consensus.cc:374] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.502641 32079 raft_consensus.cc:729] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.503012 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet cad036d297bd45d7b7c5a1f19356906f (table rebalance_test_table_11 [id=f7b46161b1f6484f9449a5d5036e221f]) successfully deleted
I20240502 08:18:04.502987 32079 consensus_queue.cc:260] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:04.502698 27258 consensus_queue.cc:237] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:04.508131 32227 ts_tablet_manager.cc:1888] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:04.508339 32227 log.cc:1196] T dd887cbb35c74e54a43d3dc6a2dde817 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/dd887cbb35c74e54a43d3dc6a2dde817
I20240502 08:18:04.509184 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet dd887cbb35c74e54a43d3dc6a2dde817 (table rebalance_test_table_3 [id=cc946163c76846418f7f80018ed16ce5]) successfully deleted
I20240502 08:18:04.508904 30157 raft_consensus.cc:2932] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:04.512887 32079 ts_tablet_manager.cc:1398] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.067s user 0.001s sys 0.001s
I20240502 08:18:04.513324 32079 tablet_bootstrap.cc:492] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:04.515511 30288 raft_consensus.cc:1261] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:04.516310 31680 consensus_queue.cc:1035] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:04.517547 27370 tablet_service.cc:1514] Processing DeleteTablet for tablet 09aa1b5d99114b3095be08c9547fbb54 with delete_type TABLET_DATA_TOMBSTONED (TS 83e171a5e16c4008923f1d43fc5ce7b1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40868
I20240502 08:18:04.517670 27258 raft_consensus.cc:1261] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.519030 27389 raft_consensus.cc:1261] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.520752 32258 tablet_replica.cc:330] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:04.521144 32258 raft_consensus.cc:2229] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.521214 32227 tablet_replica.cc:330] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:18:04.521593 32258 raft_consensus.cc:2258] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.522358 32227 raft_consensus.cc:2229] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.523829 32254 ts_tablet_manager.cc:1888] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:04.524137 32227 raft_consensus.cc:2258] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.523840 31680 raft_consensus.cc:2932] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:04.524336 32254 log.cc:1196] T 26545a868bd6415c9c9db4869b2841c7 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/26545a868bd6415c9c9db4869b2841c7
I20240502 08:18:04.524752 30288 raft_consensus.cc:2932] T e67c99e57a73443da13527ed33326a4e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:04.525328 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet 26545a868bd6415c9c9db4869b2841c7 (table rebalancer_extra_table_10 [id=9df4632350e6444089d25e0f410be77a]) successfully deleted
I20240502 08:18:04.526193 32227 ts_tablet_manager.cc:1875] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:04.528141 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e67c99e57a73443da13527ed33326a4e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:04.525224 32254 tablet_replica.cc:330] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:04.529112 32254 raft_consensus.cc:2229] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.529953 32254 raft_consensus.cc:2258] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.531188 27110 catalog_manager.cc:5484] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:04.535714 32253 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:04.539449 27518 raft_consensus.cc:1261] T e67c99e57a73443da13527ed33326a4e P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:04.540132 30138 tablet_service.cc:1514] Processing DeleteTablet for tablet e67c99e57a73443da13527ed33326a4e with delete_type TABLET_DATA_TOMBSTONED (TS 84cda20738d2452da166e31fc6da57a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:04.540409 32261 tablet_replica.cc:330] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:18:04.540618 32261 raft_consensus.cc:2229] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.540974 32261 raft_consensus.cc:2258] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.541780 32258 ts_tablet_manager.cc:1875] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.542423 27648 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:04.544957 27389 raft_consensus.cc:1261] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:04.545430 32261 ts_tablet_manager.cc:1875] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:04.548605 32079 tablet_bootstrap.cc:654] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.549902 31359 consensus_queue.cc:1035] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.551827 27109 catalog_manager.cc:5484] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:04.553071 31680 consensus_queue.cc:1035] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:04.557878 27519 raft_consensus.cc:2932] T e67c99e57a73443da13527ed33326a4e P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:04.558095 27258 raft_consensus.cc:2932] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:04.559250 32254 ts_tablet_manager.cc:1875] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.560632 32044 raft_consensus.cc:482] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:04.561486 32261 ts_tablet_manager.cc:1888] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:04.561609 32261 log.cc:1196] T 09aa1b5d99114b3095be08c9547fbb54 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/09aa1b5d99114b3095be08c9547fbb54
I20240502 08:18:04.564913 32227 ts_tablet_manager.cc:1888] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:04.565065 32227 log.cc:1196] T af2f11471c064f48924191fc2607d31c P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/af2f11471c064f48924191fc2607d31c
I20240502 08:18:04.565793 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet af2f11471c064f48924191fc2607d31c (table rebalance_test_table_1 [id=33e20eb07c524d63b8e99e7957467d8b]) successfully deleted
I20240502 08:18:04.560835 32044 raft_consensus.cc:504] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.562966 27111 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:04.568104 32044 leader_election.cc:288] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:04.571732 27648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ac899c820a43e4ab57454e224edc83" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:04.572013 27648 raft_consensus.cc:2454] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:04.572784 30226 leader_election.cc:302] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:04.573230 32044 raft_consensus.cc:2781] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:04.573398 32044 raft_consensus.cc:482] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:04.573503 32044 raft_consensus.cc:3037] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.576996 32044 raft_consensus.cc:504] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.579653 32031 raft_consensus.cc:482] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:04.579860 32031 raft_consensus.cc:504] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:04.580691 32031 leader_election.cc:288] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:04.579380 27648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ac899c820a43e4ab57454e224edc83" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:04.581658 27648 raft_consensus.cc:3037] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.582226 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039b1aea12ba4627882b603336070773" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:04.582460 27652 raft_consensus.cc:2454] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:04.583616 30226 leader_election.cc:302] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:04.583969 27500 tablet_service.cc:1467] Processing CreateTablet for tablet bbfbd57e32ca42dfb224b09c07ec28ff (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0e3b95d8f01a40d58d98de7f642d4fd4]), partition=
I20240502 08:18:04.584221 32031 raft_consensus.cc:2781] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:04.584316 32031 raft_consensus.cc:482] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:04.584390 32031 raft_consensus.cc:3037] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.584465 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbfbd57e32ca42dfb224b09c07ec28ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.587112 32031 raft_consensus.cc:504] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:04.588932 30268 tablet_service.cc:1467] Processing CreateTablet for tablet bbfbd57e32ca42dfb224b09c07ec28ff (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0e3b95d8f01a40d58d98de7f642d4fd4]), partition=
I20240502 08:18:04.588946 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039b1aea12ba4627882b603336070773" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:04.589126 27652 raft_consensus.cc:3037] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.591593 27652 raft_consensus.cc:2454] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:04.592073 30226 leader_election.cc:302] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:04.592356 31920 raft_consensus.cc:2781] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:04.594101 32179 raft_consensus.cc:1050] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad to VOTER
I20240502 08:18:04.594146 32246 raft_consensus.cc:1050] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 to VOTER
I20240502 08:18:04.594924 32246 consensus_queue.cc:237] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:04.594933 32179 consensus_queue.cc:237] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:04.597520 27648 raft_consensus.cc:2454] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:04.597875 30226 leader_election.cc:302] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:04.598273 31920 raft_consensus.cc:686] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:18:04.598716 32266 raft_consensus.cc:2781] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:04.598651 31920 consensus_queue.cc:237] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:04.599885 32031 leader_election.cc:288] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:04.600672 32044 leader_election.cc:288] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:04.601023 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbfbd57e32ca42dfb224b09c07ec28ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.602618 27109 catalog_manager.cc:5484] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7). New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:04.605446 32266 raft_consensus.cc:686] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:18:04.605768 32216 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:04.605779 32266 consensus_queue.cc:237] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.608213 31962 tablet_bootstrap.cc:492] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:04.613974 27519 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:04.614202 27111 catalog_manager.cc:5484] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:04.616645 30421 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:04.618042 30423 raft_consensus.cc:1261] T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.619222 32182 consensus_queue.cc:1035] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.623585 27648 raft_consensus.cc:1261] T 6978aa3955504c548778066ddd0910e5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.624270 32175 consensus_queue.cc:1035] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.627332 27516 raft_consensus.cc:1261] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.627482 31359 consensus_queue.cc:1035] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:04.567736 30026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ac899c820a43e4ab57454e224edc83" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:04.584014 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039b1aea12ba4627882b603336070773" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:04.628407 32175 consensus_queue.cc:1035] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.628516 30025 raft_consensus.cc:2454] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:04.586728 30006 tablet_service.cc:1467] Processing CreateTablet for tablet bbfbd57e32ca42dfb224b09c07ec28ff (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0e3b95d8f01a40d58d98de7f642d4fd4]), partition=
I20240502 08:18:04.629084 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbfbd57e32ca42dfb224b09c07ec28ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.630671 27652 raft_consensus.cc:1261] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.588290 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039b1aea12ba4627882b603336070773" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:04.601186 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ac899c820a43e4ab57454e224edc83" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:04.634136 30028 raft_consensus.cc:3037] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.634760 32246 consensus_queue.cc:1035] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.636754 30028 raft_consensus.cc:2454] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
W20240502 08:18:04.640403 30223 leader_election.cc:341] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:18:04.640972 30223 leader_election.cc:341] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:04.642618 30289 raft_consensus.cc:1261] T a6dce110666a4a038f5f46d48a3ec7af P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.643122 31359 consensus_queue.cc:1035] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.653208 32152 consensus_queue.cc:1035] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.654259 31962 tablet_bootstrap.cc:654] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.659497 32257 ts_tablet_manager.cc:1888] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:04.662894 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 09aa1b5d99114b3095be08c9547fbb54 (table rebalance_test_table_9 [id=a65036830fa64a22b9f4efd6d5bd4f9f]) successfully deleted
I20240502 08:18:04.670145 32152 consensus_queue.cc:1035] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:04.692936 32182 raft_consensus.cc:2932] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:04.698889 27648 raft_consensus.cc:2932] T 6978aa3955504c548778066ddd0910e5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:04.701829 27388 raft_consensus.cc:1261] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:04.704479 32244 tablet_copy_client.cc:308] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:18:04.705758 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet a793eaaeba89458baf8af7caf3f24c70 from peer 84cda20738d2452da166e31fc6da57a6 ({username='slave'} at 127.20.207.6:52973)
I20240502 08:18:04.705894 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet a793eaaeba89458baf8af7caf3f24c70 from peer 84cda20738d2452da166e31fc6da57a6 at {username='slave'} at 127.20.207.6:52973: session id = 84cda20738d2452da166e31fc6da57a6-a793eaaeba89458baf8af7caf3f24c70
I20240502 08:18:04.707191 27110 catalog_manager.cc:5484] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:04.708859 27278 tablet_copy_source_session.cc:205] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:04.711333 27258 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:04.712805 31962 tablet_bootstrap.cc:492] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:04.713016 31962 ts_tablet_manager.cc:1367] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.105s user 0.004s sys 0.013s
I20240502 08:18:04.713881 31962 raft_consensus.cc:348] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:04.714180 31962 raft_consensus.cc:374] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.714278 31962 raft_consensus.cc:729] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.714543 31962 consensus_queue.cc:260] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:04.715188 31962 ts_tablet_manager.cc:1398] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:04.721033 27257 consensus_queue.cc:237] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:04.721366 30423 raft_consensus.cc:2932] T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:04.730572 32246 consensus_queue.cc:1035] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:04.731854 27519 raft_consensus.cc:2932] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:04.734920 32244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a793eaaeba89458baf8af7caf3f24c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.735111 32258 ts_tablet_manager.cc:1888] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:04.735244 32258 log.cc:1196] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/cb66fb2351f442c4bd3137274cdb5449
I20240502 08:18:04.739465 32269 tablet_bootstrap.cc:492] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:04.741767 32269 tablet_bootstrap.cc:654] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.743175 32258 ts_tablet_manager.cc:1909] T cb66fb2351f442c4bd3137274cdb5449 P 84cda20738d2452da166e31fc6da57a6: Deleting consensus metadata
I20240502 08:18:04.744242 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet cb66fb2351f442c4bd3137274cdb5449 (table rebalancer_dynamic_table_new_name [id=6336a199bdbc48389a3a983413491684]) successfully deleted
I20240502 08:18:04.746835 32258 tablet_replica.cc:330] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:04.747977 32258 raft_consensus.cc:2229] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.748998 32258 raft_consensus.cc:2258] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.753322 32258 ts_tablet_manager.cc:1875] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:04.755358 30422 raft_consensus.cc:1261] T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:04.758176 32179 consensus_queue.cc:1035] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:04.754491 32244 tablet_copy_client.cc:783] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:04.760514 27110 catalog_manager.cc:5484] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7). New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:04.761194 32244 tablet_copy_client.cc:647] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:04.764643 32244 tablet_copy_client.cc:523] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:04.765465 32182 raft_consensus.cc:2932] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:04.768560 27388 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:04.769534 32244 tablet_bootstrap.cc:492] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:04.769554 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6978aa3955504c548778066ddd0910e5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:04.772687 32257 log.cc:1196] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/cb66fb2351f442c4bd3137274cdb5449
I20240502 08:18:04.772552 30422 raft_consensus.cc:2932] T 6978aa3955504c548778066ddd0910e5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:04.774639 31359 raft_consensus.cc:2932] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:04.778192 27110 catalog_manager.cc:5484] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:04.782783 27108 catalog_manager.cc:5484] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:04.786016 27650 raft_consensus.cc:1261] T 6978aa3955504c548778066ddd0910e5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:04.786450 30289 raft_consensus.cc:2932] T a6dce110666a4a038f5f46d48a3ec7af P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:04.787127 32179 consensus_queue.cc:1035] T 6978aa3955504c548778066ddd0910e5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:04.789000 27650 raft_consensus.cc:2932] T 6978aa3955504c548778066ddd0910e5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:04.790792 27389 raft_consensus.cc:2932] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:04.793569 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet 6978aa3955504c548778066ddd0910e5 with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:04.795369 32242 tablet_replica.cc:330] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:04.795570 32242 raft_consensus.cc:2229] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.795783 32242 raft_consensus.cc:2258] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.795967 27650 raft_consensus.cc:2932] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:04.796797 32079 tablet_bootstrap.cc:492] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:04.796856 32266 raft_consensus.cc:482] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:04.796967 32079 ts_tablet_manager.cc:1367] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.284s user 0.002s sys 0.002s
I20240502 08:18:04.797013 32266 raft_consensus.cc:504] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:04.797755 32266 leader_election.cc:288] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:04.797668 32079 raft_consensus.cc:348] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.797888 32079 raft_consensus.cc:374] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.797976 32079 raft_consensus.cc:729] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.798274 32079 consensus_queue.cc:260] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.799093 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbfbd57e32ca42dfb224b09c07ec28ff" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:04.799005 32079 ts_tablet_manager.cc:1398] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:04.799547 32079 tablet_bootstrap.cc:492] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:04.799578 27518 consensus_queue.cc:237] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
W20240502 08:18:04.799770 30223 leader_election.cc:341] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:04.800168 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbfbd57e32ca42dfb224b09c07ec28ff" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:18:04.801290 32242 ts_tablet_manager.cc:1875] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 08:18:04.801463 30226 leader_election.cc:341] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:04.801582 30226 leader_election.cc:302] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4; no voters: 1f0bf4bc3c5146539146b825b048bff3, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:04.801923 32266 raft_consensus.cc:2726] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:04.809074 32242 ts_tablet_manager.cc:1888] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:04.809263 32242 log.cc:1196] T 6978aa3955504c548778066ddd0910e5 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/6978aa3955504c548778066ddd0910e5
I20240502 08:18:04.809655 32254 ts_tablet_manager.cc:1888] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:04.809772 32254 log.cc:1196] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/cb66fb2351f442c4bd3137274cdb5449
I20240502 08:18:04.810180 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 6978aa3955504c548778066ddd0910e5 (table rebalance_test_table_7 [id=83c398ec83c74ebf84113a0334649e63]) successfully deleted
I20240502 08:18:04.811295 27650 raft_consensus.cc:1261] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:04.811825 32246 consensus_queue.cc:1035] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:04.815065 32257 ts_tablet_manager.cc:1909] T cb66fb2351f442c4bd3137274cdb5449 P 1f0bf4bc3c5146539146b825b048bff3: Deleting consensus metadata
I20240502 08:18:04.817847 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet cb66fb2351f442c4bd3137274cdb5449 (table rebalancer_dynamic_table_new_name [id=6336a199bdbc48389a3a983413491684]) successfully deleted
I20240502 08:18:04.818063 32258 ts_tablet_manager.cc:1888] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:04.818265 32258 log.cc:1196] T e67c99e57a73443da13527ed33326a4e P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/e67c99e57a73443da13527ed33326a4e
I20240502 08:18:04.818158 31705 raft_consensus.cc:2932] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:04.819159 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet e67c99e57a73443da13527ed33326a4e (table rebalancer_extra_table_4 [id=767cc24e34dd43bc8e4de07a278f8b27]) successfully deleted
I20240502 08:18:04.818929 27652 raft_consensus.cc:2932] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:04.819530 32254 ts_tablet_manager.cc:1909] T cb66fb2351f442c4bd3137274cdb5449 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting consensus metadata
I20240502 08:18:04.820581 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet cb66fb2351f442c4bd3137274cdb5449 (table rebalancer_dynamic_table_new_name [id=6336a199bdbc48389a3a983413491684]) successfully deleted
I20240502 08:18:04.825193 27111 catalog_manager.cc:5484] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 5 to 6, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:04.834251 27516 consensus_queue.cc:237] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:04.837291 32224 consensus_queue.cc:1035] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:04.838387 27325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.1:38505, user_credentials={real_user=slave}} blocked reactor thread for 63940.2us
I20240502 08:18:04.839169 27390 raft_consensus.cc:2932] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
W20240502 08:18:04.845981 27453 consensus_peers.cc:485] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c233aeb1e58418ba7b01f48917fabad. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:04.847340 27648 raft_consensus.cc:1261] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:04.847839 31359 consensus_queue.cc:1035] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:04.848829 30288 raft_consensus.cc:1261] T a6dce110666a4a038f5f46d48a3ec7af P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:04.849474 31359 consensus_queue.cc:1035] T a6dce110666a4a038f5f46d48a3ec7af P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:04.850764 30288 raft_consensus.cc:2932] T a6dce110666a4a038f5f46d48a3ec7af P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:04.857416 27388 raft_consensus.cc:1261] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:04.857952 31359 consensus_queue.cc:1035] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:04.858129 32079 tablet_bootstrap.cc:654] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.827291 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a6dce110666a4a038f5f46d48a3ec7af with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:04.871996 27370 tablet_service.cc:1514] Processing DeleteTablet for tablet a6dce110666a4a038f5f46d48a3ec7af with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 6)) from {username='slave'} at 127.0.0.1:40868
I20240502 08:18:04.872351 32261 tablet_replica.cc:330] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:18:04.872458 32229 raft_consensus.cc:482] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:04.872557 32261 raft_consensus.cc:2229] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.872606 32229 raft_consensus.cc:504] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:04.872886 32261 raft_consensus.cc:2258] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.873113 32269 tablet_bootstrap.cc:492] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:04.873392 32229 leader_election.cc:288] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:04.873699 27648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ac899c820a43e4ab57454e224edc83" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:04.873872 32261 ts_tablet_manager.cc:1875] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:04.873987 27648 raft_consensus.cc:2379] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f0bf4bc3c5146539146b825b048bff3 in current term 1: Already voted for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in this term.
I20240502 08:18:04.877087 31359 raft_consensus.cc:2932] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:04.879308 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ac899c820a43e4ab57454e224edc83" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:04.882174 29961 leader_election.cc:302] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, bf8ac71b15f740a0a928911360773442
I20240502 08:18:04.882505 32230 raft_consensus.cc:3037] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.883752 32079 tablet_bootstrap.cc:492] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:04.883904 32079 ts_tablet_manager.cc:1367] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.084s user 0.001s sys 0.002s
I20240502 08:18:04.885363 32230 raft_consensus.cc:2726] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:04.886204 27648 raft_consensus.cc:2932] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:04.886937 27112 catalog_manager.cc:5484] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:04.893672 32079 raft_consensus.cc:348] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:04.902073 32079 raft_consensus.cc:374] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.902271 32079 raft_consensus.cc:729] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.902644 32079 consensus_queue.cc:260] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:04.903357 32079 ts_tablet_manager.cc:1398] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20240502 08:18:04.903411 32244 tablet_bootstrap.cc:492] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:04.903878 32244 tablet_bootstrap.cc:492] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6: Bootstrap complete.
I20240502 08:18:04.904284 32244 ts_tablet_manager.cc:1367] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.135s user 0.054s sys 0.000s
I20240502 08:18:04.905251 32244 raft_consensus.cc:348] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:04.905812 32244 raft_consensus.cc:729] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: LEARNER
I20240502 08:18:04.906107 32244 consensus_queue.cc:260] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:04.906822 32244 ts_tablet_manager.cc:1398] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:04.908437 32261 ts_tablet_manager.cc:1888] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:04.908614 32261 log.cc:1196] T a6dce110666a4a038f5f46d48a3ec7af P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/a6dce110666a4a038f5f46d48a3ec7af
I20240502 08:18:04.909277 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session 84cda20738d2452da166e31fc6da57a6-a793eaaeba89458baf8af7caf3f24c70 received from {username='slave'} at 127.20.207.6:52973
I20240502 08:18:04.909372 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet a6dce110666a4a038f5f46d48a3ec7af (table rebalance_test_table_15 [id=f2de4023ae664751b94f83ff4434cd92]) successfully deleted
I20240502 08:18:04.909451 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session 84cda20738d2452da166e31fc6da57a6-a793eaaeba89458baf8af7caf3f24c70 on tablet a793eaaeba89458baf8af7caf3f24c70 with peer 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:04.913704 32269 ts_tablet_manager.cc:1367] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.174s user 0.006s sys 0.000s
I20240502 08:18:04.915562 27388 raft_consensus.cc:2932] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:04.916182 32269 raft_consensus.cc:348] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:04.923791 32269 raft_consensus.cc:374] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.924059 32269 raft_consensus.cc:729] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.924607 32269 consensus_queue.cc:260] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:04.925355 32269 ts_tablet_manager.cc:1398] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 08:18:04.944083 27388 consensus_queue.cc:237] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
W20240502 08:18:04.946753 27323 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Couldn't send request to peer 1f0bf4bc3c5146539146b825b048bff3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6cf739b314d84db08bc4cb32b0e5b0f4. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:04.946754 27516 raft_consensus.cc:1261] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:04.947258 27258 raft_consensus.cc:1261] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:04.947849 31848 consensus_queue.cc:1035] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:04.948181 32152 consensus_queue.cc:1035] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:04.962710 32224 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:04.963375 27516 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:04.963429 27258 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:04.969877 31359 raft_consensus.cc:482] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:04.970086 31359 raft_consensus.cc:504] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:04.969906 27110 catalog_manager.cc:5484] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:04.970722 31359 leader_election.cc:288] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:04.973148 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbfbd57e32ca42dfb224b09c07ec28ff" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:04.973219 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbfbd57e32ca42dfb224b09c07ec28ff" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:04.973392 30027 raft_consensus.cc:2454] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:18:04.973445 30289 raft_consensus.cc:2454] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:18:04.974053 27453 leader_election.cc:302] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:04.974684 31359 raft_consensus.cc:2781] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:04.974839 31359 raft_consensus.cc:482] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:04.974951 31359 raft_consensus.cc:3037] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.979321 31359 raft_consensus.cc:504] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:04.980486 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbfbd57e32ca42dfb224b09c07ec28ff" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:04.980110 31359 leader_election.cc:288] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:04.980691 30027 raft_consensus.cc:3037] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.980715 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbfbd57e32ca42dfb224b09c07ec28ff" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:04.980923 30288 raft_consensus.cc:3037] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.983263 30027 raft_consensus.cc:2454] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:18:04.984019 30288 raft_consensus.cc:2454] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:18:04.984647 27453 leader_election.cc:302] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:04.985232 31359 raft_consensus.cc:2781] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:04.985383 31359 raft_consensus.cc:686] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:18:04.985741 31359 consensus_queue.cc:237] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:04.987192 32307 ts_tablet_manager.cc:902] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:05.000528 27112 catalog_manager.cc:5484] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:05.005263 32307 tablet_copy_client.cc:308] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:05.006021 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet 5e0313e85d444e098bfa35be89a3ad97 from peer 1f0bf4bc3c5146539146b825b048bff3 ({username='slave'} at 127.20.207.5:54501)
I20240502 08:18:05.006186 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet 5e0313e85d444e098bfa35be89a3ad97 from peer 1f0bf4bc3c5146539146b825b048bff3 at {username='slave'} at 127.20.207.5:54501: session id = 1f0bf4bc3c5146539146b825b048bff3-5e0313e85d444e098bfa35be89a3ad97
I20240502 08:18:05.008715 27541 tablet_copy_source_session.cc:205] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:05.009902 32307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0313e85d444e098bfa35be89a3ad97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.015573 32307 tablet_copy_client.cc:783] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:05.015790 32307 tablet_copy_client.cc:647] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:05.018379 32307 tablet_copy_client.cc:523] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:05.022765 32307 tablet_bootstrap.cc:492] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:05.055781 32267 consensus_queue.cc:1035] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:18:05.063321 27388 consensus_queue.cc:237] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:05.065066 30422 raft_consensus.cc:1261] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:05.065794 32219 consensus_queue.cc:1035] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:05.067081 27258 raft_consensus.cc:1261] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:05.067669 31848 consensus_queue.cc:1035] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:05.068940 32224 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:05.069396 32266 consensus_queue.cc:1035] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:05.072706 27258 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:05.074184 30423 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:05.080509 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 04a0a58eb52f41bf818e5102996d09e0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:05.082589 27112 catalog_manager.cc:5484] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:05.096362 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet 04a0a58eb52f41bf818e5102996d09e0 with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:05.097105 32242 tablet_replica.cc:330] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:05.097342 32242 raft_consensus.cc:2229] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:05.097581 32242 raft_consensus.cc:2258] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.099676 32267 consensus_queue.cc:1035] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:05.100919 32242 ts_tablet_manager.cc:1875] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:05.101035 32307 tablet_bootstrap.cc:492] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:05.101521 32307 tablet_bootstrap.cc:492] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap complete.
I20240502 08:18:05.101804 32307 ts_tablet_manager.cc:1367] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.079s user 0.046s sys 0.006s
I20240502 08:18:05.102654 32307 raft_consensus.cc:348] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:05.102836 32307 raft_consensus.cc:729] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: LEARNER
I20240502 08:18:05.102988 32307 consensus_queue.cc:260] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:05.103413 32307 ts_tablet_manager.cc:1398] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:05.104010 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 1f0bf4bc3c5146539146b825b048bff3-5e0313e85d444e098bfa35be89a3ad97 received from {username='slave'} at 127.20.207.5:54501
I20240502 08:18:05.104161 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 1f0bf4bc3c5146539146b825b048bff3-5e0313e85d444e098bfa35be89a3ad97 on tablet 5e0313e85d444e098bfa35be89a3ad97 with peer 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:18:05.112895 32266 consensus_queue.cc:1035] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:05.113631 32242 ts_tablet_manager.cc:1888] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:05.113770 32242 log.cc:1196] T 04a0a58eb52f41bf818e5102996d09e0 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/04a0a58eb52f41bf818e5102996d09e0
I20240502 08:18:05.118758 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 04a0a58eb52f41bf818e5102996d09e0 (table rebalance_test_table_3 [id=cc946163c76846418f7f80018ed16ce5]) successfully deleted
I20240502 08:18:05.140867 27112 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:05.159644 30288 raft_consensus.cc:1261] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:05.160233 32253 consensus_queue.cc:1035] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:05.170094 30027 raft_consensus.cc:1261] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:05.170848 32253 consensus_queue.cc:1035] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:05.183667 27516 consensus_queue.cc:237] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:05.185770 27388 raft_consensus.cc:1261] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:05.186331 31359 consensus_queue.cc:1035] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:05.187422 30028 raft_consensus.cc:1261] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:05.188500 32253 consensus_queue.cc:1035] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:05.190253 31705 raft_consensus.cc:2932] T 0bea60d096f9408c9f4a857f8824d6a2 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:05.191591 27388 raft_consensus.cc:2932] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:05.191823 30028 raft_consensus.cc:2932] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:05.201472 27110 catalog_manager.cc:5484] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:05.207715 27112 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet bbfbd57e32ca42dfb224b09c07ec28ff (table rebalancer_dynamic_table [id=0e3b95d8f01a40d58d98de7f642d4fd4]). Expected version 1 got 0
I20240502 08:18:05.208580 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bea60d096f9408c9f4a857f8824d6a2 with delete_type TABLET_DATA_TOMBSTONED (TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39166
I20240502 08:18:05.211328 32358 tablet_replica.cc:330] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:18:05.211582 32358 raft_consensus.cc:2229] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:05.211849 32358 raft_consensus.cc:2258] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.213707 27112 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet bbfbd57e32ca42dfb224b09c07ec28ff (table rebalancer_dynamic_table [id=0e3b95d8f01a40d58d98de7f642d4fd4]). Expected version 1 got 0
I20240502 08:18:05.220340 32358 ts_tablet_manager.cc:1875] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:05.225749 32359 tablet.cc:1722] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:05.226940 32359 tablet.cc:1620] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:05.230638 27110 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet bbfbd57e32ca42dfb224b09c07ec28ff (table rebalancer_dynamic_table [id=0e3b95d8f01a40d58d98de7f642d4fd4]). Expected version 1 got 0
I20240502 08:18:05.237214 32361 tablet.cc:1722] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:05.237923 32361 tablet.cc:1620] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:05.238749 32362 ts_tablet_manager.cc:902] T 7c233aeb1e58418ba7b01f48917fabad P 49c5c8f70e494edd9fcda40cc0af6cc4: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:05.246629 27112 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet bbfbd57e32ca42dfb224b09c07ec28ff (table rebalancer_dynamic_table [id=0e3b95d8f01a40d58d98de7f642d4fd4]). Expected version 1 got 0
I20240502 08:18:05.248227 32358 ts_tablet_manager.cc:1888] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:05.248384 32358 log.cc:1196] T 0bea60d096f9408c9f4a857f8824d6a2 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/0bea60d096f9408c9f4a857f8824d6a2
I20240502 08:18:05.248637 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=0e3b95d8f01a40d58d98de7f642d4fd4] alter complete (version 1)
I20240502 08:18:05.249944 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet 0bea60d096f9408c9f4a857f8824d6a2 (table rebalance_test_table_5 [id=5c5af31071894d4f803992d5173e0429]) successfully deleted
I20240502 08:18:05.253227 32362 tablet_copy_client.cc:308] T 7c233aeb1e58418ba7b01f48917fabad P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:05.253979 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet 7c233aeb1e58418ba7b01f48917fabad from peer 49c5c8f70e494edd9fcda40cc0af6cc4 ({username='slave'} at 127.20.207.7:58763)
I20240502 08:18:05.254261 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet 7c233aeb1e58418ba7b01f48917fabad from peer 49c5c8f70e494edd9fcda40cc0af6cc4 at {username='slave'} at 127.20.207.7:58763: session id = 49c5c8f70e494edd9fcda40cc0af6cc4-7c233aeb1e58418ba7b01f48917fabad
I20240502 08:18:05.256273 32357 tablet.cc:1722] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:05.257401 27541 tablet_copy_source_session.cc:205] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:05.256852 32357 tablet.cc:1620] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59: MemRowSet was empty: no flush needed.
I20240502 08:18:05.259047 32362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c233aeb1e58418ba7b01f48917fabad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.259589 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0bea60d096f9408c9f4a857f8824d6a2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:05.276401 32362 tablet_copy_client.cc:783] T 7c233aeb1e58418ba7b01f48917fabad P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:05.277266 32362 tablet_copy_client.cc:647] T 7c233aeb1e58418ba7b01f48917fabad P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:05.279186 32362 tablet_copy_client.cc:523] T 7c233aeb1e58418ba7b01f48917fabad P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:05.282438 32362 tablet_bootstrap.cc:492] T 7c233aeb1e58418ba7b01f48917fabad P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:05.323563 30157 raft_consensus.cc:1203] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:05.332130 32362 tablet_bootstrap.cc:492] T 7c233aeb1e58418ba7b01f48917fabad P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:05.332753 32362 tablet_bootstrap.cc:492] T 7c233aeb1e58418ba7b01f48917fabad P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap complete.
I20240502 08:18:05.333129 32362 ts_tablet_manager.cc:1367] T 7c233aeb1e58418ba7b01f48917fabad P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.051s user 0.042s sys 0.004s
I20240502 08:18:05.334174 32362 raft_consensus.cc:348] T 7c233aeb1e58418ba7b01f48917fabad P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:05.334553 32362 raft_consensus.cc:729] T 7c233aeb1e58418ba7b01f48917fabad P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: LEARNER
I20240502 08:18:05.334841 32362 consensus_queue.cc:260] T 7c233aeb1e58418ba7b01f48917fabad P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:05.335708 32362 ts_tablet_manager.cc:1398] T 7c233aeb1e58418ba7b01f48917fabad P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:05.339638 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-7c233aeb1e58418ba7b01f48917fabad received from {username='slave'} at 127.20.207.7:58763
I20240502 08:18:05.339752 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-7c233aeb1e58418ba7b01f48917fabad on tablet 7c233aeb1e58418ba7b01f48917fabad with peer 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:05.366175 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:05.388394 32357 tablet.cc:1722] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:05.388981 32357 tablet.cc:1620] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59: MemRowSet was empty: no flush needed.
I20240502 08:18:05.392876 32361 tablet.cc:1722] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:05.393355 32361 tablet.cc:1620] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:05.398797 32359 tablet.cc:1722] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:05.399246 32359 tablet.cc:1620] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:05.410657 27108 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=0e3b95d8f01a40d58d98de7f642d4fd4] alter complete (version 2)
I20240502 08:18:05.424432 27110 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:05.435999 27370 tablet_service.cc:1467] Processing CreateTablet for tablet 47b5771de7034e10acf2f40d0ac3c4d3 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=abcec94585574b2a928dda8297060d6c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.436472 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b5771de7034e10acf2f40d0ac3c4d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.436668 30006 tablet_service.cc:1467] Processing CreateTablet for tablet 47b5771de7034e10acf2f40d0ac3c4d3 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=abcec94585574b2a928dda8297060d6c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.437108 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b5771de7034e10acf2f40d0ac3c4d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.441776 27500 tablet_service.cc:1467] Processing CreateTablet for tablet 47b5771de7034e10acf2f40d0ac3c4d3 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=abcec94585574b2a928dda8297060d6c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.442216 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b5771de7034e10acf2f40d0ac3c4d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.445631 30007 tablet_service.cc:1467] Processing CreateTablet for tablet ff41957b81c64ebf8572233937191f6f (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=abcec94585574b2a928dda8297060d6c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.446060 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff41957b81c64ebf8572233937191f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.446288 30268 tablet_service.cc:1467] Processing CreateTablet for tablet d9f41c94eaf04ac5ad77d3124f6368c9 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=abcec94585574b2a928dda8297060d6c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.446789 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f41c94eaf04ac5ad77d3124f6368c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.447062 27631 tablet_service.cc:1467] Processing CreateTablet for tablet d9f41c94eaf04ac5ad77d3124f6368c9 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=abcec94585574b2a928dda8297060d6c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.447470 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f41c94eaf04ac5ad77d3124f6368c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.450593 30138 tablet_service.cc:1467] Processing CreateTablet for tablet ff41957b81c64ebf8572233937191f6f (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=abcec94585574b2a928dda8297060d6c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.450995 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff41957b81c64ebf8572233937191f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.452299 27238 tablet_service.cc:1467] Processing CreateTablet for tablet d9f41c94eaf04ac5ad77d3124f6368c9 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=abcec94585574b2a928dda8297060d6c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.452661 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f41c94eaf04ac5ad77d3124f6368c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.459455 30400 tablet_service.cc:1467] Processing CreateTablet for tablet ff41957b81c64ebf8572233937191f6f (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=abcec94585574b2a928dda8297060d6c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.459869 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff41957b81c64ebf8572233937191f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.485188 32395 tablet_bootstrap.cc:492] T ff41957b81c64ebf8572233937191f6f P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:05.487111 32395 tablet_bootstrap.cc:654] T ff41957b81c64ebf8572233937191f6f P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.489584 32394 tablet_bootstrap.cc:492] T ff41957b81c64ebf8572233937191f6f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:05.491490 32394 tablet_bootstrap.cc:654] T ff41957b81c64ebf8572233937191f6f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.493054 32393 tablet_bootstrap.cc:492] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:05.495054 32393 tablet_bootstrap.cc:654] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.496740 32396 tablet_bootstrap.cc:492] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:05.498793 32396 tablet_bootstrap.cc:654] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.500844 32395 tablet_bootstrap.cc:492] T ff41957b81c64ebf8572233937191f6f P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:05.501046 32395 ts_tablet_manager.cc:1367] T ff41957b81c64ebf8572233937191f6f P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.004s
I20240502 08:18:05.501955 32395 raft_consensus.cc:348] T ff41957b81c64ebf8572233937191f6f P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:05.502274 32395 raft_consensus.cc:374] T ff41957b81c64ebf8572233937191f6f P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.502418 32395 raft_consensus.cc:729] T ff41957b81c64ebf8572233937191f6f P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.504109 32395 consensus_queue.cc:260] T ff41957b81c64ebf8572233937191f6f P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:05.508430 32395 ts_tablet_manager.cc:1398] T ff41957b81c64ebf8572233937191f6f P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:18:05.513068 32394 tablet_bootstrap.cc:492] T ff41957b81c64ebf8572233937191f6f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:05.513310 32394 ts_tablet_manager.cc:1367] T ff41957b81c64ebf8572233937191f6f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240502 08:18:05.514200 32394 raft_consensus.cc:348] T ff41957b81c64ebf8572233937191f6f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:05.514452 32394 raft_consensus.cc:374] T ff41957b81c64ebf8572233937191f6f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.514602 32394 raft_consensus.cc:729] T ff41957b81c64ebf8572233937191f6f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.514897 32394 consensus_queue.cc:260] T ff41957b81c64ebf8572233937191f6f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:05.516312 27112 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:05.518834 32394 ts_tablet_manager.cc:1398] T ff41957b81c64ebf8572233937191f6f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:05.523924 32393 tablet_bootstrap.cc:492] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:05.524121 32393 ts_tablet_manager.cc:1367] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240502 08:18:05.524879 32393 raft_consensus.cc:348] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:05.525137 32393 raft_consensus.cc:374] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.525274 32393 raft_consensus.cc:729] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.525558 32393 consensus_queue.cc:260] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:05.526389 30027 raft_consensus.cc:1203] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:05.533074 32393 ts_tablet_manager.cc:1398] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 08:18:05.533567 32395 tablet_bootstrap.cc:492] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:05.538637 32395 tablet_bootstrap.cc:654] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.538849 32357 tablet.cc:1722] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:05.539680 32359 tablet.cc:1722] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:05.540612 32396 tablet_bootstrap.cc:492] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:05.540812 32396 ts_tablet_manager.cc:1367] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20240502 08:18:05.541635 32396 raft_consensus.cc:348] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:05.541913 32396 raft_consensus.cc:374] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.542088 32396 raft_consensus.cc:729] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.542340 32396 consensus_queue.cc:260] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:05.542930 32396 ts_tablet_manager.cc:1398] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:05.562108 32361 tablet.cc:1722] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:05.571877 27112 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=0e3b95d8f01a40d58d98de7f642d4fd4] alter complete (version 3)
I20240502 08:18:05.572115 32307 ts_tablet_manager.cc:902] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3: Initiating tablet copy from peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:05.585337 32175 raft_consensus.cc:482] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:05.585515 32175 raft_consensus.cc:504] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:05.586191 32175 leader_election.cc:288] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:05.587407 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f41c94eaf04ac5ad77d3124f6368c9" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:05.587517 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f41c94eaf04ac5ad77d3124f6368c9" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
W20240502 08:18:05.588241 27194 leader_election.cc:341] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:18:05.588449 27191 leader_election.cc:341] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:05.588606 27191 leader_election.cc:302] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, bf8ac71b15f740a0a928911360773442
I20240502 08:18:05.589272 32179 raft_consensus.cc:2726] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:05.623490 32400 tablet_bootstrap.cc:492] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:05.625346 32400 tablet_bootstrap.cc:654] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.627353 32307 tablet_copy_client.cc:308] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.2:46291
I20240502 08:18:05.628248 27410 tablet_copy_service.cc:140] P 83e171a5e16c4008923f1d43fc5ce7b1: Received BeginTabletCopySession request for tablet 6cf739b314d84db08bc4cb32b0e5b0f4 from peer 1f0bf4bc3c5146539146b825b048bff3 ({username='slave'} at 127.20.207.5:49101)
I20240502 08:18:05.628417 27410 tablet_copy_service.cc:161] P 83e171a5e16c4008923f1d43fc5ce7b1: Beginning new tablet copy session on tablet 6cf739b314d84db08bc4cb32b0e5b0f4 from peer 1f0bf4bc3c5146539146b825b048bff3 at {username='slave'} at 127.20.207.5:49101: session id = 1f0bf4bc3c5146539146b825b048bff3-6cf739b314d84db08bc4cb32b0e5b0f4
I20240502 08:18:05.631224 27410 tablet_copy_source_session.cc:205] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:05.637099 32307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf739b314d84db08bc4cb32b0e5b0f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.646653 32399 tablet_bootstrap.cc:492] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:05.648567 32399 tablet_bootstrap.cc:654] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.654898 32397 tablet_bootstrap.cc:492] T 47b5771de7034e10acf2f40d0ac3c4d3 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:05.656776 32397 tablet_bootstrap.cc:654] T 47b5771de7034e10acf2f40d0ac3c4d3 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.644889 32307 tablet_copy_client.cc:783] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:05.657794 32307 tablet_copy_client.cc:647] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:05.664011 32307 tablet_copy_client.cc:523] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:05.670032 32400 tablet_bootstrap.cc:492] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:05.671646 32400 ts_tablet_manager.cc:1367] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.001s
I20240502 08:18:05.673306 32400 raft_consensus.cc:348] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:05.673558 32399 tablet_bootstrap.cc:492] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:18:05.674201 32399 ts_tablet_manager.cc:1367] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240502 08:18:05.674423 32392 tablet_bootstrap.cc:492] T d9f41c94eaf04ac5ad77d3124f6368c9 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:05.675215 32399 raft_consensus.cc:348] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:05.675134 32307 tablet_bootstrap.cc:492] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:05.676614 32395 tablet_bootstrap.cc:492] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:05.676779 32395 ts_tablet_manager.cc:1367] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.143s user 0.037s sys 0.036s
I20240502 08:18:05.676987 32400 raft_consensus.cc:374] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.677695 32395 raft_consensus.cc:348] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:05.677943 32395 raft_consensus.cc:374] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.678072 32395 raft_consensus.cc:729] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.678383 32395 consensus_queue.cc:260] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:05.679085 32395 ts_tablet_manager.cc:1398] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:18:05.680240 30289 raft_consensus.cc:1203] T 7c233aeb1e58418ba7b01f48917fabad P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:05.694885 27108 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:05.695111 27108 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:05.699981 27108 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet bbfbd57e32ca42dfb224b09c07ec28ff
I20240502 08:18:05.701293 30007 tablet_service.cc:1514] Processing DeleteTablet for tablet bbfbd57e32ca42dfb224b09c07ec28ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:05.701962 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet bbfbd57e32ca42dfb224b09c07ec28ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:05.707193 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet bbfbd57e32ca42dfb224b09c07ec28ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:05.708370 32397 tablet_bootstrap.cc:492] T 47b5771de7034e10acf2f40d0ac3c4d3 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:05.709609 32397 ts_tablet_manager.cc:1367] T 47b5771de7034e10acf2f40d0ac3c4d3 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.000s
I20240502 08:18:05.711019 32400 raft_consensus.cc:729] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.711367 32399 raft_consensus.cc:374] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.711828 32400 consensus_queue.cc:260] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:05.712201 32399 raft_consensus.cc:729] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.712859 32399 consensus_queue.cc:260] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:05.713516 32397 raft_consensus.cc:348] T 47b5771de7034e10acf2f40d0ac3c4d3 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:05.719506 32392 tablet_bootstrap.cc:654] T d9f41c94eaf04ac5ad77d3124f6368c9 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.719651 32397 raft_consensus.cc:374] T 47b5771de7034e10acf2f40d0ac3c4d3 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.724448 32397 raft_consensus.cc:729] T 47b5771de7034e10acf2f40d0ac3c4d3 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.731140 32400 ts_tablet_manager.cc:1398] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.046s user 0.000s sys 0.002s
I20240502 08:18:05.731595 32399 ts_tablet_manager.cc:1398] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.057s user 0.002s sys 0.000s
I20240502 08:18:05.721901 32428 tablet_replica.cc:330] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:05.732096 32428 raft_consensus.cc:2229] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:05.732359 32428 raft_consensus.cc:2258] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.720887 32429 tablet_replica.cc:330] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:05.733080 32429 raft_consensus.cc:2229] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:05.733263 32428 ts_tablet_manager.cc:1875] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.733733 32429 raft_consensus.cc:2258] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.735879 32429 ts_tablet_manager.cc:1875] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.740952 32428 ts_tablet_manager.cc:1888] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:05.741117 32428 log.cc:1196] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/bbfbd57e32ca42dfb224b09c07ec28ff
I20240502 08:18:05.741772 32428 ts_tablet_manager.cc:1909] T bbfbd57e32ca42dfb224b09c07ec28ff P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting consensus metadata
I20240502 08:18:05.745746 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet bbfbd57e32ca42dfb224b09c07ec28ff (table rebalancer_dynamic_table_new_name [id=0e3b95d8f01a40d58d98de7f642d4fd4]) successfully deleted
I20240502 08:18:05.753625 32429 ts_tablet_manager.cc:1888] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:05.753753 32429 log.cc:1196] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/bbfbd57e32ca42dfb224b09c07ec28ff
I20240502 08:18:05.758816 32429 ts_tablet_manager.cc:1909] T bbfbd57e32ca42dfb224b09c07ec28ff P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:18:05.759786 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet bbfbd57e32ca42dfb224b09c07ec28ff (table rebalancer_dynamic_table_new_name [id=0e3b95d8f01a40d58d98de7f642d4fd4]) successfully deleted
I20240502 08:18:05.770196 32397 consensus_queue.cc:260] T 47b5771de7034e10acf2f40d0ac3c4d3 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:05.775032 32069 raft_consensus.cc:482] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:05.775192 32069 raft_consensus.cc:504] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:05.775909 32069 leader_election.cc:288] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:05.778250 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff41957b81c64ebf8572233937191f6f" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:05.778458 30027 raft_consensus.cc:2454] T ff41957b81c64ebf8572233937191f6f P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:05.780961 30093 leader_election.cc:302] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:05.781317 32069 raft_consensus.cc:2781] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:05.781440 32069 raft_consensus.cc:482] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:05.781518 32069 raft_consensus.cc:3037] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.782928 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff41957b81c64ebf8572233937191f6f" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:05.783169 30422 raft_consensus.cc:2454] T ff41957b81c64ebf8572233937191f6f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:05.784339 32069 raft_consensus.cc:504] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:05.784912 32069 leader_election.cc:288] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:05.785878 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff41957b81c64ebf8572233937191f6f" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:05.786239 30423 raft_consensus.cc:3037] T ff41957b81c64ebf8572233937191f6f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.789415 30423 raft_consensus.cc:2454] T ff41957b81c64ebf8572233937191f6f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:05.789844 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff41957b81c64ebf8572233937191f6f" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:05.790360 30027 raft_consensus.cc:3037] T ff41957b81c64ebf8572233937191f6f P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.796075 32219 raft_consensus.cc:482] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:05.796309 32219 raft_consensus.cc:504] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:05.797125 32219 leader_election.cc:288] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:05.798055 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5771de7034e10acf2f40d0ac3c4d3" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:05.798354 30028 raft_consensus.cc:2454] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:18:05.799126 27323 leader_election.cc:302] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:05.800057 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5771de7034e10acf2f40d0ac3c4d3" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:18:05.800349 27519 raft_consensus.cc:2454] T 47b5771de7034e10acf2f40d0ac3c4d3 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:18:05.805900 31848 raft_consensus.cc:2781] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:05.805702 27112 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:05.805992 31848 raft_consensus.cc:482] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:05.806129 31848 raft_consensus.cc:3037] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.809098 31848 raft_consensus.cc:504] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:05.809864 31848 leader_election.cc:288] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:05.811952 32427 tablet_replica.cc:330] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:05.812170 32427 raft_consensus.cc:2229] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:05.812433 32427 raft_consensus.cc:2258] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.814230 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5771de7034e10acf2f40d0ac3c4d3" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:05.817484 30028 raft_consensus.cc:3037] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.820215 30093 leader_election.cc:302] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:05.820581 32069 raft_consensus.cc:2781] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:05.821717 32307 tablet_bootstrap.cc:492] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:05.821748 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5771de7034e10acf2f40d0ac3c4d3" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:18:05.821964 27516 raft_consensus.cc:3037] T 47b5771de7034e10acf2f40d0ac3c4d3 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.827406 32307 tablet_bootstrap.cc:492] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap complete.
I20240502 08:18:05.827374 30138 tablet_service.cc:1467] Processing CreateTablet for tablet 1e3318ee67994fceb2ed0855712ef828 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=201f0cf586a741ce9eb1cbcafb1dad99]), partition=
I20240502 08:18:05.827733 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e3318ee67994fceb2ed0855712ef828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.827733 32307 ts_tablet_manager.cc:1367] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.153s user 0.061s sys 0.015s
I20240502 08:18:05.830192 32307 raft_consensus.cc:348] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:05.830507 32307 raft_consensus.cc:729] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: LEARNER
I20240502 08:18:05.830528 30400 tablet_service.cc:1467] Processing CreateTablet for tablet 1e3318ee67994fceb2ed0855712ef828 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=201f0cf586a741ce9eb1cbcafb1dad99]), partition=
I20240502 08:18:05.830832 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e3318ee67994fceb2ed0855712ef828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.830731 32307 consensus_queue.cc:260] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:05.830976 30028 raft_consensus.cc:2454] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:18:05.833014 32307 ts_tablet_manager.cc:1398] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:05.832988 27323 leader_election.cc:302] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:05.834178 32219 raft_consensus.cc:2781] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:05.834266 27410 tablet_copy_service.cc:342] P 83e171a5e16c4008923f1d43fc5ce7b1: Request end of tablet copy session 1f0bf4bc3c5146539146b825b048bff3-6cf739b314d84db08bc4cb32b0e5b0f4 received from {username='slave'} at 127.20.207.5:49101
I20240502 08:18:05.834353 32219 raft_consensus.cc:686] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:18:05.834434 27410 tablet_copy_service.cc:433] P 83e171a5e16c4008923f1d43fc5ce7b1: ending tablet copy session 1f0bf4bc3c5146539146b825b048bff3-6cf739b314d84db08bc4cb32b0e5b0f4 on tablet 6cf739b314d84db08bc4cb32b0e5b0f4 with peer 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:18:05.834700 32219 consensus_queue.cc:237] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:05.839309 27516 raft_consensus.cc:2454] T 47b5771de7034e10acf2f40d0ac3c4d3 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:18:05.842525 30027 raft_consensus.cc:2454] T ff41957b81c64ebf8572233937191f6f P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:05.845507 30007 tablet_service.cc:1467] Processing CreateTablet for tablet 1e3318ee67994fceb2ed0855712ef828 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=201f0cf586a741ce9eb1cbcafb1dad99]), partition=
I20240502 08:18:05.847770 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e3318ee67994fceb2ed0855712ef828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.851075 32394 tablet_bootstrap.cc:492] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:05.853061 32394 tablet_bootstrap.cc:654] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.856877 32397 ts_tablet_manager.cc:1398] T 47b5771de7034e10acf2f40d0ac3c4d3 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.147s user 0.002s sys 0.000s
I20240502 08:18:05.866619 32427 ts_tablet_manager.cc:1875] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.873344 32393 tablet_bootstrap.cc:492] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:05.875020 32393 tablet_bootstrap.cc:654] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.877127 32393 tablet_bootstrap.cc:492] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:05.877389 32393 ts_tablet_manager.cc:1367] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:05.878458 32393 raft_consensus.cc:348] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:05.878787 32393 raft_consensus.cc:374] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.878906 32393 raft_consensus.cc:729] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.879215 32393 consensus_queue.cc:260] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:05.888336 32393 ts_tablet_manager.cc:1398] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.011s user 0.002s sys 0.004s
I20240502 08:18:05.892505 32172 raft_consensus.cc:1050] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER 84cda20738d2452da166e31fc6da57a6 to VOTER
I20240502 08:18:05.893409 32172 consensus_queue.cc:237] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:05.916285 32394 tablet_bootstrap.cc:492] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:05.916589 32394 ts_tablet_manager.cc:1367] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.009s
I20240502 08:18:05.917505 32394 raft_consensus.cc:348] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:05.920063 32394 raft_consensus.cc:374] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.920224 32394 raft_consensus.cc:729] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.920552 32394 consensus_queue.cc:260] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:05.931344 32395 tablet_bootstrap.cc:492] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:05.932960 32395 tablet_bootstrap.cc:654] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.935561 27112 catalog_manager.cc:5484] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:05.940649 30159 raft_consensus.cc:1261] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:05.941378 32179 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:05.944025 27516 raft_consensus.cc:1261] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:05.944674 32179 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:05.945479 32395 tablet_bootstrap.cc:492] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:05.945641 32395 ts_tablet_manager.cc:1367] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20240502 08:18:05.946638 27390 raft_consensus.cc:1261] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:05.947196 32395 raft_consensus.cc:348] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:05.947453 32395 raft_consensus.cc:374] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.947543 32395 raft_consensus.cc:729] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.947783 32395 consensus_queue.cc:260] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:05.948158 32175 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:05.948926 32395 ts_tablet_manager.cc:1398] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:05.957543 32069 raft_consensus.cc:686] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:05.957902 32069 consensus_queue.cc:237] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:05.957782 32175 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:05.958711 32394 ts_tablet_manager.cc:1398] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.042s user 0.001s sys 0.001s
I20240502 08:18:05.961289 30159 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:05.963606 27516 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:05.968894 27109 catalog_manager.cc:5484] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:05.972165 30028 raft_consensus.cc:1203] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:05.980201 27111 catalog_manager.cc:5484] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:05.992321 27390 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:06.010635 32392 tablet_bootstrap.cc:492] T d9f41c94eaf04ac5ad77d3124f6368c9 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:06.010876 32392 ts_tablet_manager.cc:1367] T d9f41c94eaf04ac5ad77d3124f6368c9 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.339s user 0.027s sys 0.020s
I20240502 08:18:06.011926 32392 raft_consensus.cc:348] T d9f41c94eaf04ac5ad77d3124f6368c9 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:06.012552 32392 raft_consensus.cc:374] T d9f41c94eaf04ac5ad77d3124f6368c9 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:06.012720 32392 raft_consensus.cc:729] T d9f41c94eaf04ac5ad77d3124f6368c9 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:06.013128 32392 consensus_queue.cc:260] T d9f41c94eaf04ac5ad77d3124f6368c9 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:06.015344 32392 ts_tablet_manager.cc:1398] T d9f41c94eaf04ac5ad77d3124f6368c9 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
W20240502 08:18:06.016031 27194 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.6:38899, user_credentials={real_user=slave}} blocked reactor thread for 48558.3us
I20240502 08:18:06.027925 32427 ts_tablet_manager.cc:1888] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:06.028970 32427 log.cc:1196] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/bbfbd57e32ca42dfb224b09c07ec28ff
I20240502 08:18:06.029551 32427 ts_tablet_manager.cc:1909] T bbfbd57e32ca42dfb224b09c07ec28ff P 1f0bf4bc3c5146539146b825b048bff3: Deleting consensus metadata
I20240502 08:18:06.031071 27258 consensus_queue.cc:237] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:06.032375 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet bbfbd57e32ca42dfb224b09c07ec28ff (table rebalancer_dynamic_table_new_name [id=0e3b95d8f01a40d58d98de7f642d4fd4]) successfully deleted
I20240502 08:18:06.034653 27390 raft_consensus.cc:1261] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:06.036167 32463 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:06.038661 32467 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:06.041059 27390 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:06.042325 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a793eaaeba89458baf8af7caf3f24c70 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:06.046164 27111 catalog_manager.cc:5484] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:06.048648 30157 raft_consensus.cc:1261] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:06.049903 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet a793eaaeba89458baf8af7caf3f24c70 with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:06.050256 32429 tablet_replica.cc:330] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:06.050493 32429 raft_consensus.cc:2229] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.050750 32429 raft_consensus.cc:2258] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.052278 32429 ts_tablet_manager.cc:1875] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:06.057300 32429 ts_tablet_manager.cc:1888] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:06.057441 32429 log.cc:1196] T a793eaaeba89458baf8af7caf3f24c70 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/a793eaaeba89458baf8af7caf3f24c70
I20240502 08:18:06.058137 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet a793eaaeba89458baf8af7caf3f24c70 (table rebalance_test_table_7 [id=83c398ec83c74ebf84113a0334649e63]) successfully deleted
I20240502 08:18:06.062701 32462 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:06.066687 32069 raft_consensus.cc:482] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:06.066869 32069 raft_consensus.cc:504] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:06.067471 32069 leader_election.cc:288] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:06.067965 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3318ee67994fceb2ed0855712ef828" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:06.068197 30422 raft_consensus.cc:2454] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:06.068603 30093 leader_election.cc:302] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:06.068735 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3318ee67994fceb2ed0855712ef828" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:06.068987 30028 raft_consensus.cc:2454] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:06.069631 32356 raft_consensus.cc:1050] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 to VOTER
I20240502 08:18:06.070417 32356 consensus_queue.cc:237] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:06.071970 32069 raft_consensus.cc:2781] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:06.072064 32069 raft_consensus.cc:482] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:06.072165 32069 raft_consensus.cc:3037] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:06.074419 32069 raft_consensus.cc:504] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:06.075841 30157 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:06.078372 32069 leader_election.cc:288] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:06.079460 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3318ee67994fceb2ed0855712ef828" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:06.079684 30422 raft_consensus.cc:3037] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:06.082688 30422 raft_consensus.cc:2454] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:06.088528 30027 raft_consensus.cc:1261] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:06.089154 32356 consensus_queue.cc:1035] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 08:18:06.091301 27650 raft_consensus.cc:1261] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:06.092854 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3318ee67994fceb2ed0855712ef828" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:06.093041 30028 raft_consensus.cc:3037] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:06.093986 32253 consensus_queue.cc:1035] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:06.098089 27390 raft_consensus.cc:1261] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:06.099026 32356 consensus_queue.cc:1035] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:06.103516 30028 raft_consensus.cc:2454] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:06.103817 32253 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:06.104236 27388 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:06.095552 30093 leader_election.cc:302] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:06.106083 27650 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:06.108448 27258 consensus_queue.cc:237] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:06.108736 30028 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:06.112035 27109 catalog_manager.cc:5484] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:06.113188 31753 raft_consensus.cc:482] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:06.113363 31753 raft_consensus.cc:504] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:06.114014 31753 leader_election.cc:288] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:06.114751 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f41c94eaf04ac5ad77d3124f6368c9" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:06.115005 30288 raft_consensus.cc:2454] T d9f41c94eaf04ac5ad77d3124f6368c9 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:18:06.115490 27584 leader_election.cc:302] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:06.115861 31753 raft_consensus.cc:2781] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:06.115960 31753 raft_consensus.cc:482] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:06.116075 31753 raft_consensus.cc:3037] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:06.116035 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f41c94eaf04ac5ad77d3124f6368c9" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:06.116292 27257 raft_consensus.cc:2454] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:18:06.118858 31753 raft_consensus.cc:504] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:06.121145 32437 raft_consensus.cc:2781] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:06.121292 32437 raft_consensus.cc:686] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:06.121569 32437 consensus_queue.cc:237] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:06.124569 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f41c94eaf04ac5ad77d3124f6368c9" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:06.124768 27257 raft_consensus.cc:3037] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:06.125402 31753 leader_election.cc:288] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:06.126752 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f41c94eaf04ac5ad77d3124f6368c9" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:06.127005 30288 raft_consensus.cc:3037] T d9f41c94eaf04ac5ad77d3124f6368c9 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:06.129799 27257 raft_consensus.cc:2454] T d9f41c94eaf04ac5ad77d3124f6368c9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:06.133023 27586 leader_election.cc:302] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:06.134914 31753 raft_consensus.cc:2781] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:06.135115 31753 raft_consensus.cc:686] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:18:06.135422 31753 consensus_queue.cc:237] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:06.138525 27516 consensus_queue.cc:237] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:06.138945 27650 raft_consensus.cc:1261] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:06.138994 27519 raft_consensus.cc:1261] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:06.139555 32463 consensus_queue.cc:1035] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:06.140897 32463 consensus_queue.cc:1035] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:06.141569 27388 raft_consensus.cc:1261] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:06.142791 32356 consensus_queue.cc:1035] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:06.144665 30028 raft_consensus.cc:1261] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:06.146382 31705 consensus_queue.cc:1035] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:06.146991 31705 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:06.147397 30288 raft_consensus.cc:2454] T d9f41c94eaf04ac5ad77d3124f6368c9 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:06.150724 27388 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:06.157652 27109 catalog_manager.cc:5484] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:06.160342 30027 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:06.165128 27109 catalog_manager.cc:5484] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:06.166121 32175 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
W20240502 08:18:06.166752 27191 consensus_peers.cc:485] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Couldn't send request to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20a836dd442943d08a87d61afc785cd1. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:06.170306 27519 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:06.171185 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5e0313e85d444e098bfa35be89a3ad97 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:06.175474 27116 catalog_manager.cc:5484] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:06.176050 27111 catalog_manager.cc:5484] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:06.179019 27650 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:06.192569 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0313e85d444e098bfa35be89a3ad97 with delete_type TABLET_DATA_TOMBSTONED (TS bf8ac71b15f740a0a928911360773442 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44788
I20240502 08:18:06.212474 32480 tablet_replica.cc:330] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:18:06.212774 32480 raft_consensus.cc:2229] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.213210 32480 raft_consensus.cc:2258] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.214349 32480 ts_tablet_manager.cc:1875] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:06.224539 32480 ts_tablet_manager.cc:1888] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:06.224661 32480 log.cc:1196] T 5e0313e85d444e098bfa35be89a3ad97 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/5e0313e85d444e098bfa35be89a3ad97
I20240502 08:18:06.225461 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 5e0313e85d444e098bfa35be89a3ad97 (table rebalance_test_table_9 [id=a65036830fa64a22b9f4efd6d5bd4f9f]) successfully deleted
I20240502 08:18:06.230494 27516 consensus_queue.cc:237] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:06.233103 27650 raft_consensus.cc:1261] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:06.233439 30288 raft_consensus.cc:1261] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:06.234050 32253 consensus_queue.cc:1035] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:06.234330 32356 consensus_queue.cc:1035] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:06.235991 27453 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 -> Peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Couldn't send request to peer 1f0bf4bc3c5146539146b825b048bff3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e2b9488fe8ac46099b7e8adf4d0b9cd5. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:06.237952 32356 raft_consensus.cc:2932] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:06.238636 27650 raft_consensus.cc:2932] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:06.241197 30288 raft_consensus.cc:2932] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:06.246253 27110 catalog_manager.cc:5484] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index -1 to 2, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:06.257362 27516 consensus_queue.cc:237] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:06.259621 27258 raft_consensus.cc:1261] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:06.259631 27650 raft_consensus.cc:1261] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:18:06.259836 27453 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 903502e57ab2430383807b0d46a39847. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:06.260169 32253 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:06.260422 32356 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:06.263319 31705 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:06.264109 27258 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:06.268016 32224 consensus_queue.cc:1035] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:06.269537 32219 consensus_queue.cc:1035] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:06.282474 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:06.285725 27109 catalog_manager.cc:5484] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:06.289047 32253 raft_consensus.cc:1050] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 to VOTER
I20240502 08:18:06.289855 32253 consensus_queue.cc:237] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:06.320124 27388 raft_consensus.cc:1261] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:06.320761 30025 raft_consensus.cc:1261] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:06.321022 32499 consensus_queue.cc:1035] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:06.321254 30288 raft_consensus.cc:1261] T 7c233aeb1e58418ba7b01f48917fabad P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:06.321503 32437 consensus_queue.cc:1035] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:06.321869 32499 consensus_queue.cc:1035] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:06.320659 27650 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:06.332703 30422 raft_consensus.cc:1261] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:06.333560 32437 consensus_queue.cc:1035] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:06.335568 27652 raft_consensus.cc:1261] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:06.337270 32497 consensus_queue.cc:1035] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:06.361877 32437 consensus_queue.cc:1035] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:06.370451 32444 consensus_queue.cc:1035] T ff41957b81c64ebf8572233937191f6f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:06.375108 27110 catalog_manager.cc:5576] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899) does not have the latest schema for tablet 1e3318ee67994fceb2ed0855712ef828 (table rebalancer_dynamic_table [id=201f0cf586a741ce9eb1cbcafb1dad99]). Expected version 1 got 0
I20240502 08:18:06.383816 27116 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet 1e3318ee67994fceb2ed0855712ef828 (table rebalancer_dynamic_table [id=201f0cf586a741ce9eb1cbcafb1dad99]). Expected version 1 got 0
I20240502 08:18:06.384296 32499 raft_consensus.cc:2932] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:06.392839 27648 raft_consensus.cc:2932] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:06.393419 27388 raft_consensus.cc:2932] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:06.395960 27116 catalog_manager.cc:5484] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:06.398723 30288 raft_consensus.cc:2932] T 7c233aeb1e58418ba7b01f48917fabad P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:06.404709 27519 consensus_queue.cc:237] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:06.413400 30289 raft_consensus.cc:1261] T 7c233aeb1e58418ba7b01f48917fabad P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:06.415135 32500 consensus_queue.cc:1035] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:06.415634 27648 raft_consensus.cc:1261] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:06.415722 32521 tablet.cc:1722] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:06.416121 32500 consensus_queue.cc:1035] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:06.416287 32521 tablet.cc:1620] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:18:06.418395 32499 raft_consensus.cc:2932] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:06.422168 27112 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=201f0cf586a741ce9eb1cbcafb1dad99] alter complete (version 1)
I20240502 08:18:06.422080 30289 raft_consensus.cc:2932] T 7c233aeb1e58418ba7b01f48917fabad P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:06.422565 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7c233aeb1e58418ba7b01f48917fabad with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:06.424793 27112 catalog_manager.cc:5484] T 7c233aeb1e58418ba7b01f48917fabad P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 5 to 6, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:06.424691 27650 raft_consensus.cc:2932] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:06.428417 27370 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c233aeb1e58418ba7b01f48917fabad with delete_type TABLET_DATA_TOMBSTONED (TS 83e171a5e16c4008923f1d43fc5ce7b1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40868
I20240502 08:18:06.434044 32522 tablet.cc:1722] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:06.440217 27112 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet 1e3318ee67994fceb2ed0855712ef828 (table rebalancer_dynamic_table [id=201f0cf586a741ce9eb1cbcafb1dad99]). Expected version 1 got 0
I20240502 08:18:06.441354 27110 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet 1e3318ee67994fceb2ed0855712ef828 (table rebalancer_dynamic_table [id=201f0cf586a741ce9eb1cbcafb1dad99]). Expected version 1 got 0
I20240502 08:18:06.441395 32522 tablet.cc:1620] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:06.444626 32525 tablet.cc:1722] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:06.445184 32525 tablet.cc:1620] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:06.453763 27388 consensus_queue.cc:237] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:06.456198 30025 raft_consensus.cc:1261] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:06.456524 27326 consensus_peers.cc:485] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f63eec7e63fe4e71b1958903e0d2cb00. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:06.457259 32224 consensus_queue.cc:1035] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:06.458675 30288 raft_consensus.cc:1261] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:06.459452 32224 consensus_queue.cc:1035] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:06.461737 32524 tablet_replica.cc:330] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:18:06.461970 32524 raft_consensus.cc:2229] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.462414 32524 raft_consensus.cc:2258] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.462785 31848 raft_consensus.cc:2932] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:06.463300 30288 raft_consensus.cc:2932] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:06.463478 30025 raft_consensus.cc:2932] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:06.465171 32524 ts_tablet_manager.cc:1875] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:06.469262 27110 catalog_manager.cc:5484] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:06.484779 32524 ts_tablet_manager.cc:1888] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:06.493608 32524 log.cc:1196] T 7c233aeb1e58418ba7b01f48917fabad P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/7c233aeb1e58418ba7b01f48917fabad
I20240502 08:18:06.494515 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 7c233aeb1e58418ba7b01f48917fabad (table rebalance_test_table_3 [id=cc946163c76846418f7f80018ed16ce5]) successfully deleted
I20240502 08:18:06.502959 30157 consensus_queue.cc:237] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:06.505875 30025 raft_consensus.cc:1261] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:06.505975 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d33916984034ab99f957026b3e30d38. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:06.505976 30288 raft_consensus.cc:1261] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:06.506330 32444 consensus_queue.cc:1035] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:06.506762 32444 consensus_queue.cc:1035] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:06.508848 32444 raft_consensus.cc:2932] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:06.510640 30025 raft_consensus.cc:2932] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:06.511065 30288 raft_consensus.cc:2932] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:06.514607 27110 catalog_manager.cc:5484] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:06.530916 31848 raft_consensus.cc:1050] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1: attempting to promote NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 to VOTER
I20240502 08:18:06.531754 31848 consensus_queue.cc:237] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:06.532085 27112 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:06.534694 27258 raft_consensus.cc:1261] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:06.535634 27516 raft_consensus.cc:1261] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:06.536363 32219 consensus_queue.cc:1035] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:06.536506 30025 raft_consensus.cc:1261] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:06.536628 32224 consensus_queue.cc:1035] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:06.539850 32521 tablet.cc:1722] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:06.539911 32224 consensus_queue.cc:1035] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:06.540325 32521 tablet.cc:1620] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:18:06.541473 32522 tablet.cc:1722] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:06.541905 32522 tablet.cc:1620] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:06.546449 32525 tablet.cc:1722] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:06.546866 32525 tablet.cc:1620] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:06.558708 32224 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:06.562243 30027 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:06.562687 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=201f0cf586a741ce9eb1cbcafb1dad99] alter complete (version 2)
I20240502 08:18:06.566977 27258 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:06.572284 27110 catalog_manager.cc:5484] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:06.581869 27516 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:06.593464 27390 consensus_queue.cc:237] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:06.595729 30027 raft_consensus.cc:1261] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:06.596441 27519 raft_consensus.cc:1261] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:06.596419 32553 consensus_queue.cc:1035] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20240502 08:18:06.599759 32219 consensus_queue.cc:1035] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:06.600407 32549 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:06.601078 30027 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:06.603453 27516 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:06.607646 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6cf739b314d84db08bc4cb32b0e5b0f4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:06.609005 27110 catalog_manager.cc:5484] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 5 to 6, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:06.614634 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf739b314d84db08bc4cb32b0e5b0f4 with delete_type TABLET_DATA_TOMBSTONED (TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39166
I20240502 08:18:06.621306 32559 tablet_replica.cc:330] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:18:06.621582 32559 raft_consensus.cc:2229] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.622179 32559 raft_consensus.cc:2258] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.623229 32559 ts_tablet_manager.cc:1875] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:06.628724 31753 consensus_queue.cc:1035] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:06.629374 32559 ts_tablet_manager.cc:1888] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:06.629510 32559 log.cc:1196] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/6cf739b314d84db08bc4cb32b0e5b0f4
I20240502 08:18:06.634639 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet 6cf739b314d84db08bc4cb32b0e5b0f4 (table rebalance_test_table_11 [id=f7b46161b1f6484f9449a5d5036e221f]) successfully deleted
I20240502 08:18:06.648466 32473 consensus_queue.cc:1035] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:06.660791 32560 ts_tablet_manager.cc:902] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:06.682646 27116 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:06.693603 32521 tablet.cc:1722] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:06.695350 32522 tablet.cc:1722] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:06.695427 32525 tablet.cc:1722] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:06.704403 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=201f0cf586a741ce9eb1cbcafb1dad99] alter complete (version 3)
I20240502 08:18:06.710489 32560 tablet_copy_client.cc:308] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:18:06.711213 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet 20a836dd442943d08a87d61afc785cd1 from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad ({username='slave'} at 127.20.207.8:40413)
I20240502 08:18:06.711380 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet 20a836dd442943d08a87d61afc785cd1 from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad at {username='slave'} at 127.20.207.8:40413: session id = 31a1e33dcdbf42f58ea72ce1c0afb0ad-20a836dd442943d08a87d61afc785cd1
I20240502 08:18:06.723196 27278 tablet_copy_source_session.cc:205] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:06.724493 32560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a836dd442943d08a87d61afc785cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.730139 32560 tablet_copy_client.cc:783] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:06.730386 32560 tablet_copy_client.cc:647] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:06.732177 32560 tablet_copy_client.cc:523] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:06.735134 32560 tablet_bootstrap.cc:492] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:06.778976 32560 tablet_bootstrap.cc:492] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:06.779419 32560 tablet_bootstrap.cc:492] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap complete.
I20240502 08:18:06.779742 32560 ts_tablet_manager.cc:1367] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.045s user 0.033s sys 0.010s
I20240502 08:18:06.780488 32560 raft_consensus.cc:348] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:06.780741 32560 raft_consensus.cc:729] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: LEARNER
I20240502 08:18:06.780920 32560 consensus_queue.cc:260] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:06.782966 32560 ts_tablet_manager.cc:1398] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:18:06.783546 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-20a836dd442943d08a87d61afc785cd1 received from {username='slave'} at 127.20.207.8:40413
I20240502 08:18:06.783660 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-20a836dd442943d08a87d61afc785cd1 on tablet 20a836dd442943d08a87d61afc785cd1 with peer 31a1e33dcdbf42f58ea72ce1c0afb0ad
I20240502 08:18:06.802114 32592 ts_tablet_manager.cc:902] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:06.808724 27112 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:06.808936 27112 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:06.812613 27112 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 1e3318ee67994fceb2ed0855712ef828
I20240502 08:18:06.813777 30007 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e3318ee67994fceb2ed0855712ef828 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:06.813906 30138 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e3318ee67994fceb2ed0855712ef828 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:06.814184 30400 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e3318ee67994fceb2ed0855712ef828 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:06.817163 27112 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:06.824678 32593 ts_tablet_manager.cc:902] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:06.829525 30140 tablet_service.cc:1467] Processing CreateTablet for tablet 50f9cce760b24f31a85bdcda52178f9f (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=f18ddbbe6f0440debd82be537390727a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:06.830121 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50f9cce760b24f31a85bdcda52178f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.830425 32598 tablet_replica.cc:330] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:06.830471 30006 tablet_service.cc:1467] Processing CreateTablet for tablet 50f9cce760b24f31a85bdcda52178f9f (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=f18ddbbe6f0440debd82be537390727a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:06.830624 32598 raft_consensus.cc:2229] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:06.830875 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50f9cce760b24f31a85bdcda52178f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.831037 32598 raft_consensus.cc:2258] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.831848 27370 tablet_service.cc:1467] Processing CreateTablet for tablet 45a661d705a9438394315be30674e1f2 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=f18ddbbe6f0440debd82be537390727a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:06.832247 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45a661d705a9438394315be30674e1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.835119 30401 tablet_service.cc:1467] Processing CreateTablet for tablet 50f9cce760b24f31a85bdcda52178f9f (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=f18ddbbe6f0440debd82be537390727a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:06.835580 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50f9cce760b24f31a85bdcda52178f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.837153 30268 tablet_service.cc:1467] Processing CreateTablet for tablet de4b822c4aca4f7f92b73cc9279ddec5 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=f18ddbbe6f0440debd82be537390727a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:06.837169 27499 tablet_service.cc:1467] Processing CreateTablet for tablet 45a661d705a9438394315be30674e1f2 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=f18ddbbe6f0440debd82be537390727a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:06.837540 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de4b822c4aca4f7f92b73cc9279ddec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.837604 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45a661d705a9438394315be30674e1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.840319 27500 tablet_service.cc:1467] Processing CreateTablet for tablet de4b822c4aca4f7f92b73cc9279ddec5 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=f18ddbbe6f0440debd82be537390727a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:06.840739 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de4b822c4aca4f7f92b73cc9279ddec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.845567 30270 tablet_service.cc:1467] Processing CreateTablet for tablet 45a661d705a9438394315be30674e1f2 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=f18ddbbe6f0440debd82be537390727a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:06.845896 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45a661d705a9438394315be30674e1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.868271 32602 tablet_bootstrap.cc:492] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:06.876505 32604 tablet_bootstrap.cc:492] T 50f9cce760b24f31a85bdcda52178f9f P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:06.877455 32606 tablet_bootstrap.cc:492] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:06.877777 32605 tablet_bootstrap.cc:492] T 45a661d705a9438394315be30674e1f2 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:06.871299 30138 tablet_service.cc:1467] Processing CreateTablet for tablet de4b822c4aca4f7f92b73cc9279ddec5 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=f18ddbbe6f0440debd82be537390727a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:06.879439 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de4b822c4aca4f7f92b73cc9279ddec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.880697 32606 tablet_bootstrap.cc:654] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:06.882166 32605 tablet_bootstrap.cc:654] T 45a661d705a9438394315be30674e1f2 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:06.883502 32604 tablet_bootstrap.cc:654] T 50f9cce760b24f31a85bdcda52178f9f P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:06.886713 32601 tablet_bootstrap.cc:492] T 50f9cce760b24f31a85bdcda52178f9f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:06.888651 32601 tablet_bootstrap.cc:654] T 50f9cce760b24f31a85bdcda52178f9f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:06.889324 32606 tablet_bootstrap.cc:492] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:06.890246 32602 tablet_bootstrap.cc:654] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:06.890775 32606 ts_tablet_manager.cc:1367] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 08:18:06.891834 32607 ts_tablet_manager.cc:902] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59: Initiating tablet copy from peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:06.893237 32606 raft_consensus.cc:348] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:06.896071 32606 raft_consensus.cc:374] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:06.896435 32606 raft_consensus.cc:729] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:06.897006 32606 consensus_queue.cc:260] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:06.898988 32599 tablet_replica.cc:330] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:06.899168 32606 ts_tablet_manager.cc:1398] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:18:06.899437 32606 tablet_bootstrap.cc:492] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:06.899423 32599 raft_consensus.cc:2229] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.899812 32599 raft_consensus.cc:2258] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.900620 32604 tablet_bootstrap.cc:492] T 50f9cce760b24f31a85bdcda52178f9f P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:06.900765 32604 ts_tablet_manager.cc:1367] T 50f9cce760b24f31a85bdcda52178f9f P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20240502 08:18:06.901604 32604 raft_consensus.cc:348] T 50f9cce760b24f31a85bdcda52178f9f P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:06.903041 32606 tablet_bootstrap.cc:654] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:06.903367 32604 raft_consensus.cc:374] T 50f9cce760b24f31a85bdcda52178f9f P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:06.903515 32604 raft_consensus.cc:729] T 50f9cce760b24f31a85bdcda52178f9f P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:06.904112 32604 consensus_queue.cc:260] T 50f9cce760b24f31a85bdcda52178f9f P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:06.905303 32602 tablet_bootstrap.cc:492] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:06.905702 32602 ts_tablet_manager.cc:1367] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.001s
I20240502 08:18:06.910123 32601 tablet_bootstrap.cc:492] T 50f9cce760b24f31a85bdcda52178f9f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:06.910394 32601 ts_tablet_manager.cc:1367] T 50f9cce760b24f31a85bdcda52178f9f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240502 08:18:06.911339 32601 raft_consensus.cc:348] T 50f9cce760b24f31a85bdcda52178f9f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:06.912055 32604 ts_tablet_manager.cc:1398] T 50f9cce760b24f31a85bdcda52178f9f P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 08:18:06.915724 32605 tablet_bootstrap.cc:492] T 45a661d705a9438394315be30674e1f2 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:18:06.915936 32605 ts_tablet_manager.cc:1367] T 45a661d705a9438394315be30674e1f2 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.005s
I20240502 08:18:06.916013 27110 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:06.916949 32605 raft_consensus.cc:348] T 45a661d705a9438394315be30674e1f2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:06.917219 32605 raft_consensus.cc:374] T 45a661d705a9438394315be30674e1f2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:06.917367 32605 raft_consensus.cc:729] T 45a661d705a9438394315be30674e1f2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:18:06.917719 32603 tablet_bootstrap.cc:492] T 45a661d705a9438394315be30674e1f2 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:06.917694 32605 consensus_queue.cc:260] T 45a661d705a9438394315be30674e1f2 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:06.919561 32605 ts_tablet_manager.cc:1398] T 45a661d705a9438394315be30674e1f2 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:06.919981 32603 tablet_bootstrap.cc:654] T 45a661d705a9438394315be30674e1f2 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:06.920264 32601 raft_consensus.cc:374] T 50f9cce760b24f31a85bdcda52178f9f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:06.920390 32601 raft_consensus.cc:729] T 50f9cce760b24f31a85bdcda52178f9f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:06.920910 32601 consensus_queue.cc:260] T 50f9cce760b24f31a85bdcda52178f9f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:06.923094 32601 ts_tablet_manager.cc:1398] T 50f9cce760b24f31a85bdcda52178f9f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 08:18:06.917615 32592 tablet_copy_client.cc:308] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:06.917620 32597 tablet_replica.cc:330] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:06.924010 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet e2b9488fe8ac46099b7e8adf4d0b9cd5 from peer 1f0bf4bc3c5146539146b825b048bff3 ({username='slave'} at 127.20.207.5:54501)
I20240502 08:18:06.924176 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet e2b9488fe8ac46099b7e8adf4d0b9cd5 from peer 1f0bf4bc3c5146539146b825b048bff3 at {username='slave'} at 127.20.207.5:54501: session id = 1f0bf4bc3c5146539146b825b048bff3-e2b9488fe8ac46099b7e8adf4d0b9cd5
I20240502 08:18:06.924233 32606 tablet_bootstrap.cc:492] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:06.924417 32606 ts_tablet_manager.cc:1367] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240502 08:18:06.925210 32606 raft_consensus.cc:348] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:06.925420 32606 raft_consensus.cc:374] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:06.925508 32606 raft_consensus.cc:729] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:06.925770 32606 consensus_queue.cc:260] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:06.925920 32607 tablet_copy_client.cc:308] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.2:46291
I20240502 08:18:06.926303 32606 ts_tablet_manager.cc:1398] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:06.926837 27541 tablet_copy_source_session.cc:205] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:06.929090 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 33cc7f3146794b7f84c07e706fd26062 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6ece274098c14d158c92bfa9d98f797c]), partition=
I20240502 08:18:06.929172 27410 tablet_copy_service.cc:140] P 83e171a5e16c4008923f1d43fc5ce7b1: Received BeginTabletCopySession request for tablet f63eec7e63fe4e71b1958903e0d2cb00 from peer b01e7c8a302d41078fa39cfef665ff59 ({username='slave'} at 127.20.207.3:46497)
I20240502 08:18:06.929311 27410 tablet_copy_service.cc:161] P 83e171a5e16c4008923f1d43fc5ce7b1: Beginning new tablet copy session on tablet f63eec7e63fe4e71b1958903e0d2cb00 from peer b01e7c8a302d41078fa39cfef665ff59 at {username='slave'} at 127.20.207.3:46497: session id = b01e7c8a302d41078fa39cfef665ff59-f63eec7e63fe4e71b1958903e0d2cb00
I20240502 08:18:06.929452 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33cc7f3146794b7f84c07e706fd26062. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.932610 27410 tablet_copy_source_session.cc:205] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:06.934365 32607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63eec7e63fe4e71b1958903e0d2cb00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.925290 32597 raft_consensus.cc:2229] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.935705 32597 raft_consensus.cc:2258] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.936693 32592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2b9488fe8ac46099b7e8adf4d0b9cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.937055 32597 ts_tablet_manager.cc:1875] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.940018 30138 tablet_service.cc:1467] Processing CreateTablet for tablet 33cc7f3146794b7f84c07e706fd26062 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6ece274098c14d158c92bfa9d98f797c]), partition=
I20240502 08:18:06.940817 32606 tablet_bootstrap.cc:492] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:06.942405 32606 tablet_bootstrap.cc:654] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:06.944061 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33cc7f3146794b7f84c07e706fd26062. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.946609 32592 tablet_copy_client.cc:783] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:06.946843 32592 tablet_copy_client.cc:647] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:06.951536 32592 tablet_copy_client.cc:523] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:06.951754 32597 ts_tablet_manager.cc:1888] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:06.952775 32597 log.cc:1196] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/1e3318ee67994fceb2ed0855712ef828
I20240502 08:18:06.953442 32597 ts_tablet_manager.cc:1909] T 1e3318ee67994fceb2ed0855712ef828 P 1f0bf4bc3c5146539146b825b048bff3: Deleting consensus metadata
I20240502 08:18:06.956019 32607 tablet_copy_client.cc:783] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:06.956243 32607 tablet_copy_client.cc:647] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:06.956773 30400 tablet_service.cc:1467] Processing CreateTablet for tablet 33cc7f3146794b7f84c07e706fd26062 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6ece274098c14d158c92bfa9d98f797c]), partition=
I20240502 08:18:06.956909 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet 1e3318ee67994fceb2ed0855712ef828 (table rebalancer_dynamic_table_new_name [id=201f0cf586a741ce9eb1cbcafb1dad99]) successfully deleted
I20240502 08:18:06.957315 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33cc7f3146794b7f84c07e706fd26062. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.957768 32044 raft_consensus.cc:482] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:06.957960 32607 tablet_copy_client.cc:523] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:06.957927 32044 raft_consensus.cc:504] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:06.958668 32044 leader_election.cc:288] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:06.959506 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4b822c4aca4f7f92b73cc9279ddec5" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:18:06.959956 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4b822c4aca4f7f92b73cc9279ddec5" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
W20240502 08:18:06.960556 30226 leader_election.cc:341] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:18:06.960920 30226 leader_election.cc:341] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:06.961051 30226 leader_election.cc:302] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4; no voters: 84cda20738d2452da166e31fc6da57a6, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:06.961583 32044 raft_consensus.cc:2726] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:06.962239 32599 ts_tablet_manager.cc:1875] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.965706 32607 tablet_bootstrap.cc:492] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:06.967942 32598 ts_tablet_manager.cc:1875] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.970778 32592 tablet_bootstrap.cc:492] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:06.971179 32606 tablet_bootstrap.cc:492] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:06.971329 32606 ts_tablet_manager.cc:1367] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.005s
I20240502 08:18:06.971664 32044 raft_consensus.cc:482] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:06.971810 32044 raft_consensus.cc:504] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:06.972476 32044 leader_election.cc:288] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:06.972869 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a661d705a9438394315be30674e1f2" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:18:06.973080 27388 raft_consensus.cc:2454] T 45a661d705a9438394315be30674e1f2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:06.973270 32606 raft_consensus.cc:348] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:06.973469 32606 raft_consensus.cc:374] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:06.973592 32606 raft_consensus.cc:729] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:06.973655 30225 leader_election.cc:302] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:06.973850 32606 consensus_queue.cc:260] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:06.974184 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a661d705a9438394315be30674e1f2" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:18:06.974342 32044 raft_consensus.cc:2781] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:06.974438 32044 raft_consensus.cc:482] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:06.974529 32044 raft_consensus.cc:3037] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:18:06.974678 30226 leader_election.cc:341] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:06.976971 32606 ts_tablet_manager.cc:1398] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:18:06.978952 32044 raft_consensus.cc:504] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:06.980481 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a661d705a9438394315be30674e1f2" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
W20240502 08:18:06.981040 30226 leader_election.cc:341] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:06.981992 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a661d705a9438394315be30674e1f2" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:18:06.982303 27388 raft_consensus.cc:3037] T 45a661d705a9438394315be30674e1f2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:06.986428 27388 raft_consensus.cc:2454] T 45a661d705a9438394315be30674e1f2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:06.991680 30225 leader_election.cc:302] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1; no voters: b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:07.005167 32044 leader_election.cc:288] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:07.005512 32044 raft_consensus.cc:2781] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:07.005717 32044 raft_consensus.cc:686] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:18:07.006119 32044 consensus_queue.cc:237] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:07.012076 32592 tablet_bootstrap.cc:492] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:07.012576 32592 tablet_bootstrap.cc:492] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap complete.
I20240502 08:18:07.012846 32592 ts_tablet_manager.cc:1367] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.006s
I20240502 08:18:07.013823 32592 raft_consensus.cc:348] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:07.014159 32592 raft_consensus.cc:729] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: LEARNER
I20240502 08:18:07.014386 32592 consensus_queue.cc:260] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:07.024606 32592 ts_tablet_manager.cc:1398] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20240502 08:18:07.033649 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 1f0bf4bc3c5146539146b825b048bff3-e2b9488fe8ac46099b7e8adf4d0b9cd5 received from {username='slave'} at 127.20.207.5:54501
I20240502 08:18:07.033797 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 1f0bf4bc3c5146539146b825b048bff3-e2b9488fe8ac46099b7e8adf4d0b9cd5 on tablet e2b9488fe8ac46099b7e8adf4d0b9cd5 with peer 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:18:07.055975 32602 raft_consensus.cc:348] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:07.056242 32602 raft_consensus.cc:374] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.056397 32602 raft_consensus.cc:729] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.056674 32602 consensus_queue.cc:260] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:07.057641 32602 ts_tablet_manager.cc:1398] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.151s user 0.000s sys 0.002s
I20240502 08:18:07.058527 32602 tablet_bootstrap.cc:492] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:07.062213 32602 tablet_bootstrap.cc:654] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:07.082513 32593 tablet_copy_client.cc:308] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:07.088191 32598 ts_tablet_manager.cc:1888] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:07.088387 32598 log.cc:1196] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/1e3318ee67994fceb2ed0855712ef828
I20240502 08:18:07.088972 32598 ts_tablet_manager.cc:1909] T 1e3318ee67994fceb2ed0855712ef828 P 84cda20738d2452da166e31fc6da57a6: Deleting consensus metadata
I20240502 08:18:07.094555 32630 ts_tablet_manager.cc:902] T 0d33916984034ab99f957026b3e30d38 P b01e7c8a302d41078fa39cfef665ff59: Initiating tablet copy from peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:07.100414 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet 903502e57ab2430383807b0d46a39847 from peer 49c5c8f70e494edd9fcda40cc0af6cc4 ({username='slave'} at 127.20.207.7:58763)
I20240502 08:18:07.100641 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet 903502e57ab2430383807b0d46a39847 from peer 49c5c8f70e494edd9fcda40cc0af6cc4 at {username='slave'} at 127.20.207.7:58763: session id = 49c5c8f70e494edd9fcda40cc0af6cc4-903502e57ab2430383807b0d46a39847
I20240502 08:18:07.105962 27541 tablet_copy_source_session.cc:205] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:07.109858 32593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903502e57ab2430383807b0d46a39847. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:07.112382 27110 catalog_manager.cc:5484] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7). New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:07.118453 30225 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.207.2:46291, user_credentials={real_user=slave}} blocked reactor thread for 126825us
I20240502 08:18:07.118633 32593 tablet_copy_client.cc:783] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:07.119866 32593 tablet_copy_client.cc:647] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:07.121665 32593 tablet_copy_client.cc:523] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:07.125149 32593 tablet_bootstrap.cc:492] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:07.125962 32601 tablet_bootstrap.cc:492] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:07.126178 32603 tablet_bootstrap.cc:492] T 45a661d705a9438394315be30674e1f2 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:07.127810 32603 ts_tablet_manager.cc:1367] T 45a661d705a9438394315be30674e1f2 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.227s user 0.004s sys 0.002s
I20240502 08:18:07.128101 32601 tablet_bootstrap.cc:654] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:07.128662 32603 raft_consensus.cc:348] T 45a661d705a9438394315be30674e1f2 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:07.130542 32603 raft_consensus.cc:374] T 45a661d705a9438394315be30674e1f2 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.130710 32603 raft_consensus.cc:729] T 45a661d705a9438394315be30674e1f2 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.130980 32603 consensus_queue.cc:260] T 45a661d705a9438394315be30674e1f2 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:07.131700 32603 ts_tablet_manager.cc:1398] T 45a661d705a9438394315be30674e1f2 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:18:07.136461 32601 tablet_bootstrap.cc:492] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:07.136653 32601 ts_tablet_manager.cc:1367] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20240502 08:18:07.137424 32601 raft_consensus.cc:348] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:07.137702 32601 raft_consensus.cc:374] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.137820 32601 raft_consensus.cc:729] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.138149 32601 consensus_queue.cc:260] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:07.142194 32601 ts_tablet_manager.cc:1398] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:07.143777 32603 tablet_bootstrap.cc:492] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:07.140672 32599 ts_tablet_manager.cc:1888] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:07.146867 32599 log.cc:1196] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/1e3318ee67994fceb2ed0855712ef828
I20240502 08:18:07.147718 32599 ts_tablet_manager.cc:1909] T 1e3318ee67994fceb2ed0855712ef828 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting consensus metadata
I20240502 08:18:07.152053 30422 raft_consensus.cc:1203] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:07.154954 32448 raft_consensus.cc:482] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:07.155140 32448 raft_consensus.cc:504] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:07.155994 32448 leader_election.cc:288] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:07.157327 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50f9cce760b24f31a85bdcda52178f9f" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:07.157582 30027 raft_consensus.cc:2454] T 50f9cce760b24f31a85bdcda52178f9f P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:07.161059 30093 leader_election.cc:302] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:07.162202 32607 tablet_bootstrap.cc:492] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:07.162684 32607 tablet_bootstrap.cc:492] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap complete.
I20240502 08:18:07.162942 32607 ts_tablet_manager.cc:1367] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.197s user 0.017s sys 0.038s
I20240502 08:18:07.163736 32607 raft_consensus.cc:348] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:07.164047 32607 raft_consensus.cc:729] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: LEARNER
I20240502 08:18:07.164471 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50f9cce760b24f31a85bdcda52178f9f" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:07.164741 30422 raft_consensus.cc:2454] T 50f9cce760b24f31a85bdcda52178f9f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:07.165576 32448 raft_consensus.cc:2781] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:07.165711 32448 raft_consensus.cc:482] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:07.165833 32448 raft_consensus.cc:3037] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.165861 32603 tablet_bootstrap.cc:654] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:07.174443 32448 raft_consensus.cc:504] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:07.164333 32607 consensus_queue.cc:260] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:07.175792 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50f9cce760b24f31a85bdcda52178f9f" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:07.176038 30422 raft_consensus.cc:3037] T 50f9cce760b24f31a85bdcda52178f9f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.178954 32607 ts_tablet_manager.cc:1398] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240502 08:18:07.179489 27410 tablet_copy_service.cc:342] P 83e171a5e16c4008923f1d43fc5ce7b1: Request end of tablet copy session b01e7c8a302d41078fa39cfef665ff59-f63eec7e63fe4e71b1958903e0d2cb00 received from {username='slave'} at 127.20.207.3:46497
I20240502 08:18:07.179603 27410 tablet_copy_service.cc:433] P 83e171a5e16c4008923f1d43fc5ce7b1: ending tablet copy session b01e7c8a302d41078fa39cfef665ff59-f63eec7e63fe4e71b1958903e0d2cb00 on tablet f63eec7e63fe4e71b1958903e0d2cb00 with peer b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:07.196851 30422 raft_consensus.cc:2454] T 50f9cce760b24f31a85bdcda52178f9f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:07.197335 30093 leader_election.cc:302] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:07.197613 32444 raft_consensus.cc:2781] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:07.197743 32444 raft_consensus.cc:686] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:07.198580 32444 consensus_queue.cc:237] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:07.201424 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50f9cce760b24f31a85bdcda52178f9f" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:07.201634 30025 raft_consensus.cc:3037] T 50f9cce760b24f31a85bdcda52178f9f P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.206435 30025 raft_consensus.cc:2454] T 50f9cce760b24f31a85bdcda52178f9f P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:07.210634 32603 tablet_bootstrap.cc:492] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:07.210847 32603 ts_tablet_manager.cc:1367] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.000s
I20240502 08:18:07.211623 32603 raft_consensus.cc:348] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:07.211897 32603 raft_consensus.cc:374] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.212018 32603 raft_consensus.cc:729] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.212329 32603 consensus_queue.cc:260] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:07.214255 32603 ts_tablet_manager.cc:1398] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:07.217299 31309 raft_consensus.cc:482] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:07.217484 31309 raft_consensus.cc:504] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:07.218345 31309 leader_election.cc:288] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:07.219242 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33cc7f3146794b7f84c07e706fd26062" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:07.219489 30288 raft_consensus.cc:2454] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:07.220072 30355 leader_election.cc:302] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:18:07.221558 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33cc7f3146794b7f84c07e706fd26062" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
W20240502 08:18:07.222213 30358 leader_election.cc:341] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:07.225729 31309 raft_consensus.cc:2781] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:07.225888 31309 raft_consensus.cc:482] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:07.226020 31309 raft_consensus.cc:3037] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.230767 31309 raft_consensus.cc:504] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:07.235813 30025 raft_consensus.cc:1203] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:07.243712 32448 leader_election.cc:288] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:07.246794 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33cc7f3146794b7f84c07e706fd26062" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
W20240502 08:18:07.248149 30358 leader_election.cc:341] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:07.250804 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33cc7f3146794b7f84c07e706fd26062" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:07.251073 30288 raft_consensus.cc:3037] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.255195 30288 raft_consensus.cc:2454] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:07.255668 30355 leader_election.cc:302] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters: 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:07.260414 31309 leader_election.cc:288] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:07.263175 31309 raft_consensus.cc:2781] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:07.263595 31309 raft_consensus.cc:686] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:18:07.264200 31309 consensus_queue.cc:237] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:07.279316 27110 catalog_manager.cc:5484] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:07.281013 32496 raft_consensus.cc:482] T 45a661d705a9438394315be30674e1f2 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:07.281177 32496 raft_consensus.cc:504] T 45a661d705a9438394315be30674e1f2 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:07.281936 32496 leader_election.cc:288] T 45a661d705a9438394315be30674e1f2 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:07.282830 27390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a661d705a9438394315be30674e1f2" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:18:07.283113 27390 raft_consensus.cc:2379] T 45a661d705a9438394315be30674e1f2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b01e7c8a302d41078fa39cfef665ff59 in current term 1: Already voted for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in this term.
I20240502 08:18:07.284504 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a661d705a9438394315be30674e1f2" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:07.285163 27453 leader_election.cc:302] T 45a661d705a9438394315be30674e1f2 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:07.285657 32496 raft_consensus.cc:3037] T 45a661d705a9438394315be30674e1f2 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.289404 32496 raft_consensus.cc:2726] T 45a661d705a9438394315be30674e1f2 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:07.293988 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet 1e3318ee67994fceb2ed0855712ef828 (table rebalancer_dynamic_table_new_name [id=201f0cf586a741ce9eb1cbcafb1dad99]) successfully deleted
I20240502 08:18:07.295043 32630 tablet_copy_client.cc:308] T 0d33916984034ab99f957026b3e30d38 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.6:38899
I20240502 08:18:07.295367 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet 1e3318ee67994fceb2ed0855712ef828 (table rebalancer_dynamic_table_new_name [id=201f0cf586a741ce9eb1cbcafb1dad99]) successfully deleted
I20240502 08:18:07.295893 30180 tablet_copy_service.cc:140] P 84cda20738d2452da166e31fc6da57a6: Received BeginTabletCopySession request for tablet 0d33916984034ab99f957026b3e30d38 from peer b01e7c8a302d41078fa39cfef665ff59 ({username='slave'} at 127.20.207.3:35877)
I20240502 08:18:07.296082 30180 tablet_copy_service.cc:161] P 84cda20738d2452da166e31fc6da57a6: Beginning new tablet copy session on tablet 0d33916984034ab99f957026b3e30d38 from peer b01e7c8a302d41078fa39cfef665ff59 at {username='slave'} at 127.20.207.3:35877: session id = b01e7c8a302d41078fa39cfef665ff59-0d33916984034ab99f957026b3e30d38
I20240502 08:18:07.297353 27110 catalog_manager.cc:5484] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:07.299014 30180 tablet_copy_source_session.cc:205] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:07.300350 32602 tablet_bootstrap.cc:492] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:07.300351 32630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d33916984034ab99f957026b3e30d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:07.300607 32602 ts_tablet_manager.cc:1367] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.242s user 0.043s sys 0.093s
I20240502 08:18:07.301367 32602 raft_consensus.cc:348] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:07.301585 32602 raft_consensus.cc:374] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.301661 32602 raft_consensus.cc:729] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.301900 32602 consensus_queue.cc:260] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:07.302577 32602 ts_tablet_manager.cc:1398] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:07.307070 32630 tablet_copy_client.cc:783] T 0d33916984034ab99f957026b3e30d38 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:07.307322 32630 tablet_copy_client.cc:647] T 0d33916984034ab99f957026b3e30d38 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:07.308511 32602 tablet_bootstrap.cc:492] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:07.309109 32630 tablet_copy_client.cc:523] T 0d33916984034ab99f957026b3e30d38 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:07.310256 32602 tablet_bootstrap.cc:654] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:07.312470 32630 tablet_bootstrap.cc:492] T 0d33916984034ab99f957026b3e30d38 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:07.329142 32602 tablet_bootstrap.cc:492] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:07.329357 32602 ts_tablet_manager.cc:1367] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20240502 08:18:07.330438 32602 raft_consensus.cc:348] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:07.330714 32602 raft_consensus.cc:374] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.330868 32602 raft_consensus.cc:729] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.331190 32602 consensus_queue.cc:260] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:07.332088 32602 ts_tablet_manager.cc:1398] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:07.337677 32593 tablet_bootstrap.cc:492] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:07.338169 32593 tablet_bootstrap.cc:492] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap complete.
I20240502 08:18:07.338449 32593 ts_tablet_manager.cc:1367] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.213s user 0.063s sys 0.014s
I20240502 08:18:07.339447 32593 raft_consensus.cc:348] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:07.339712 32593 raft_consensus.cc:729] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: LEARNER
I20240502 08:18:07.339934 32593 consensus_queue.cc:260] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:07.343050 32630 tablet_bootstrap.cc:492] T 0d33916984034ab99f957026b3e30d38 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:07.343473 32630 tablet_bootstrap.cc:492] T 0d33916984034ab99f957026b3e30d38 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap complete.
I20240502 08:18:07.343710 32630 ts_tablet_manager.cc:1367] T 0d33916984034ab99f957026b3e30d38 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.023s
I20240502 08:18:07.344388 32630 raft_consensus.cc:348] T 0d33916984034ab99f957026b3e30d38 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:07.344614 32630 raft_consensus.cc:729] T 0d33916984034ab99f957026b3e30d38 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: LEARNER
I20240502 08:18:07.344650 32593 ts_tablet_manager.cc:1398] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 08:18:07.344813 32630 consensus_queue.cc:260] T 0d33916984034ab99f957026b3e30d38 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:07.345369 32630 ts_tablet_manager.cc:1398] T 0d33916984034ab99f957026b3e30d38 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:07.346117 30180 tablet_copy_service.cc:342] P 84cda20738d2452da166e31fc6da57a6: Request end of tablet copy session b01e7c8a302d41078fa39cfef665ff59-0d33916984034ab99f957026b3e30d38 received from {username='slave'} at 127.20.207.3:35877
I20240502 08:18:07.346235 30180 tablet_copy_service.cc:433] P 84cda20738d2452da166e31fc6da57a6: ending tablet copy session b01e7c8a302d41078fa39cfef665ff59-0d33916984034ab99f957026b3e30d38 on tablet 0d33916984034ab99f957026b3e30d38 with peer b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:07.347903 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-903502e57ab2430383807b0d46a39847 received from {username='slave'} at 127.20.207.7:58763
I20240502 08:18:07.348032 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-903502e57ab2430383807b0d46a39847 on tablet 903502e57ab2430383807b0d46a39847 with peer 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:07.376040 27258 consensus_queue.cc:237] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:07.378651 27516 raft_consensus.cc:1261] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:18:07.378912 27191 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Couldn't send request to peer 1f0bf4bc3c5146539146b825b048bff3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3724af3b82c43b0a1036f4d9703a3a9. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:07.379335 32462 consensus_queue.cc:1035] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:07.379922 27388 raft_consensus.cc:1261] T a3724af3b82c43b0a1036f4d9703a3a9 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:07.380481 32462 consensus_queue.cc:1035] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:07.382210 32172 raft_consensus.cc:2932] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:07.383687 27390 raft_consensus.cc:2932] T a3724af3b82c43b0a1036f4d9703a3a9 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:07.387233 27516 raft_consensus.cc:2932] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:07.391255 32496 raft_consensus.cc:482] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:07.391407 32496 raft_consensus.cc:504] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:07.391372 27109 catalog_manager.cc:5484] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:07.391993 32496 leader_election.cc:288] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:07.392756 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4b822c4aca4f7f92b73cc9279ddec5" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:07.392923 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4b822c4aca4f7f92b73cc9279ddec5" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:07.392978 30157 raft_consensus.cc:2454] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:18:07.393126 30288 raft_consensus.cc:2454] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:18:07.393977 27453 leader_election.cc:302] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:07.394899 32496 raft_consensus.cc:2781] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:07.394994 32496 raft_consensus.cc:482] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:07.395109 32496 raft_consensus.cc:3037] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.397717 32496 raft_consensus.cc:504] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:07.404057 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4b822c4aca4f7f92b73cc9279ddec5" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:07.404291 30157 raft_consensus.cc:3037] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.404340 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4b822c4aca4f7f92b73cc9279ddec5" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:07.404521 30288 raft_consensus.cc:3037] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.407202 30288 raft_consensus.cc:2454] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:18:07.407569 30157 raft_consensus.cc:2454] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:18:07.407805 27453 leader_election.cc:302] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:07.408103 32500 raft_consensus.cc:2781] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:07.408262 32500 raft_consensus.cc:686] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:18:07.408578 32500 consensus_queue.cc:237] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:07.409693 32496 leader_election.cc:288] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:07.412885 27110 catalog_manager.cc:5484] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:07.417238 30288 raft_consensus.cc:1203] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:07.429100 27516 raft_consensus.cc:1203] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:07.441591 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:07.452594 30288 raft_consensus.cc:1261] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:07.453177 32648 consensus_queue.cc:1035] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:07.460676 30157 raft_consensus.cc:3037] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.463114 30157 raft_consensus.cc:1261] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:07.463646 32648 consensus_queue.cc:1035] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:07.470489 27110 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet 33cc7f3146794b7f84c07e706fd26062 (table rebalancer_dynamic_table [id=6ece274098c14d158c92bfa9d98f797c]). Expected version 1 got 0
I20240502 08:18:07.478350 32696 tablet.cc:1722] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:07.479014 32696 tablet.cc:1620] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:07.479714 27110 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet 33cc7f3146794b7f84c07e706fd26062 (table rebalancer_dynamic_table [id=6ece274098c14d158c92bfa9d98f797c]). Expected version 1 got 0
I20240502 08:18:07.480620 32695 tablet.cc:1722] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:07.483253 32695 tablet.cc:1620] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:07.485200 27110 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet 33cc7f3146794b7f84c07e706fd26062 (table rebalancer_dynamic_table [id=6ece274098c14d158c92bfa9d98f797c]). Expected version 1 got 0
I20240502 08:18:07.490644 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=6ece274098c14d158c92bfa9d98f797c] alter complete (version 1)
I20240502 08:18:07.495594 32626 consensus_queue.cc:1035] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:07.504155 27116 catalog_manager.cc:5576] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899) does not have the latest schema for tablet 33cc7f3146794b7f84c07e706fd26062 (table rebalancer_dynamic_table [id=6ece274098c14d158c92bfa9d98f797c]). Expected version 1 got 0
I20240502 08:18:07.506660 32628 consensus_queue.cc:1035] T 45a661d705a9438394315be30674e1f2 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:07.508513 27516 raft_consensus.cc:1203] T 0d33916984034ab99f957026b3e30d38 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:07.513154 32703 tablet.cc:1722] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:07.513851 32703 tablet.cc:1620] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:18:07.598106 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:07.604581 32695 tablet.cc:1722] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:07.605139 32695 tablet.cc:1620] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:07.605424 32703 tablet.cc:1722] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:07.605862 32703 tablet.cc:1620] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:18:07.605964 32696 tablet.cc:1722] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:07.606385 32696 tablet.cc:1620] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:07.614921 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=6ece274098c14d158c92bfa9d98f797c] alter complete (version 2)
I20240502 08:18:07.618319 32496 raft_consensus.cc:1050] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 to VOTER
I20240502 08:18:07.619050 32496 consensus_queue.cc:237] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:07.623219 27648 raft_consensus.cc:1261] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:07.623265 30288 raft_consensus.cc:1261] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:07.623832 32500 consensus_queue.cc:1035] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:07.626633 30025 raft_consensus.cc:1261] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:07.627211 32717 consensus_queue.cc:1035] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:07.628641 32646 consensus_queue.cc:1035] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:07.632223 32646 raft_consensus.cc:2932] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:07.635131 30288 raft_consensus.cc:2932] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:07.643745 27109 catalog_manager.cc:5484] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 2 to 3, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:07.649507 27648 raft_consensus.cc:2932] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:07.652289 32172 raft_consensus.cc:1050] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad to VOTER
I20240502 08:18:07.653105 32172 consensus_queue.cc:237] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:07.663345 30027 raft_consensus.cc:2932] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:07.665792 27519 consensus_queue.cc:237] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:07.668452 27648 raft_consensus.cc:1261] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:07.668970 32500 consensus_queue.cc:1035] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:07.670876 30028 raft_consensus.cc:1261] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:07.671846 30422 raft_consensus.cc:1261] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:07.671835 32500 consensus_queue.cc:1035] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:07.672086 32646 raft_consensus.cc:2932] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:07.672362 32462 consensus_queue.cc:1035] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:07.673127 30027 raft_consensus.cc:2932] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:07.675952 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e2b9488fe8ac46099b7e8adf4d0b9cd5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:07.678710 27109 catalog_manager.cc:5484] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:07.679894 27519 raft_consensus.cc:1261] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:07.680387 32172 consensus_queue.cc:1035] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:07.680639 27650 raft_consensus.cc:1261] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:07.685107 32448 consensus_queue.cc:1035] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:07.691421 32462 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:07.692065 30422 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:07.693010 32172 consensus_queue.cc:1035] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:07.695617 27519 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:07.698078 27116 catalog_manager.cc:5484] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:07.699746 27650 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:07.678725 27648 raft_consensus.cc:2932] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:07.712330 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet e2b9488fe8ac46099b7e8adf4d0b9cd5 with delete_type TABLET_DATA_TOMBSTONED (TS 49c5c8f70e494edd9fcda40cc0af6cc4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:07.724838 27109 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:07.731554 32437 consensus_queue.cc:1035] T 50f9cce760b24f31a85bdcda52178f9f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:07.740612 32695 tablet.cc:1722] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:07.741379 32703 tablet.cc:1722] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:07.743585 32696 tablet.cc:1722] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:07.757889 27112 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=6ece274098c14d158c92bfa9d98f797c] alter complete (version 3)
I20240502 08:18:07.796510 32732 ts_tablet_manager.cc:902] T a3724af3b82c43b0a1036f4d9703a3a9 P 1f0bf4bc3c5146539146b825b048bff3: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:07.806447 32727 tablet_replica.cc:330] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:07.806768 32727 raft_consensus.cc:2229] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:07.808554 32721 consensus_queue.cc:1035] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:07.808761 32732 tablet_copy_client.cc:308] T a3724af3b82c43b0a1036f4d9703a3a9 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:18:07.809516 32727 raft_consensus.cc:2258] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.809609 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet a3724af3b82c43b0a1036f4d9703a3a9 from peer 1f0bf4bc3c5146539146b825b048bff3 ({username='slave'} at 127.20.207.5:51633)
I20240502 08:18:07.809736 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet a3724af3b82c43b0a1036f4d9703a3a9 from peer 1f0bf4bc3c5146539146b825b048bff3 at {username='slave'} at 127.20.207.5:51633: session id = 1f0bf4bc3c5146539146b825b048bff3-a3724af3b82c43b0a1036f4d9703a3a9
I20240502 08:18:07.814565 32727 ts_tablet_manager.cc:1875] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:07.815438 27278 tablet_copy_source_session.cc:205] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:07.817025 32732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3724af3b82c43b0a1036f4d9703a3a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:07.824905 32732 tablet_copy_client.cc:783] T a3724af3b82c43b0a1036f4d9703a3a9 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:07.825145 32732 tablet_copy_client.cc:647] T a3724af3b82c43b0a1036f4d9703a3a9 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:07.830438 32646 consensus_queue.cc:1035] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:07.834090 32732 tablet_copy_client.cc:523] T a3724af3b82c43b0a1036f4d9703a3a9 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:07.843374 32732 tablet_bootstrap.cc:492] T a3724af3b82c43b0a1036f4d9703a3a9 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:07.844796 32727 ts_tablet_manager.cc:1888] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:07.845185 32727 log.cc:1196] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/e2b9488fe8ac46099b7e8adf4d0b9cd5
I20240502 08:18:07.846318 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet e2b9488fe8ac46099b7e8adf4d0b9cd5 (table rebalancer_extra_table_0 [id=3600173badac4b5bbac03472cd151621]) successfully deleted
I20240502 08:18:07.863703 32717 raft_consensus.cc:1050] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 to VOTER
I20240502 08:18:07.863744 27109 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:07.864027 27109 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:07.864843 32717 consensus_queue.cc:237] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:07.868842 27109 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 33cc7f3146794b7f84c07e706fd26062
I20240502 08:18:07.870193 30400 tablet_service.cc:1514] Processing DeleteTablet for tablet 33cc7f3146794b7f84c07e706fd26062 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:07.870543 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet 33cc7f3146794b7f84c07e706fd26062 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:07.870851 32727 tablet_replica.cc:330] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:07.870949 30138 tablet_service.cc:1514] Processing DeleteTablet for tablet 33cc7f3146794b7f84c07e706fd26062 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:07.871094 32727 raft_consensus.cc:2229] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:07.871353 32727 raft_consensus.cc:2258] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.872298 32727 ts_tablet_manager.cc:1875] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.885018 32727 ts_tablet_manager.cc:1888] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:07.886377 32727 log.cc:1196] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/33cc7f3146794b7f84c07e706fd26062
I20240502 08:18:07.887250 32727 ts_tablet_manager.cc:1909] T 33cc7f3146794b7f84c07e706fd26062 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting consensus metadata
I20240502 08:18:07.889746 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet 33cc7f3146794b7f84c07e706fd26062 (table rebalancer_dynamic_table_new_name [id=6ece274098c14d158c92bfa9d98f797c]) successfully deleted
I20240502 08:18:07.890883 32747 tablet_replica.cc:330] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:07.891152 32747 raft_consensus.cc:2229] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:07.891394 32747 raft_consensus.cc:2258] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.898779 32745 tablet_replica.cc:330] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:07.899816 32745 raft_consensus.cc:2229] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:07.900008 30288 raft_consensus.cc:1261] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:07.900310 32745 raft_consensus.cc:2258] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.900609 32646 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:07.901264 32745 ts_tablet_manager.cc:1875] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.908082 27648 raft_consensus.cc:1261] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:07.908811 32646 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:07.909204 32745 ts_tablet_manager.cc:1888] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:07.909371 32745 log.cc:1196] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/33cc7f3146794b7f84c07e706fd26062
I20240502 08:18:07.909950 32745 ts_tablet_manager.cc:1909] T 33cc7f3146794b7f84c07e706fd26062 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting consensus metadata
I20240502 08:18:07.912135 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet 33cc7f3146794b7f84c07e706fd26062 (table rebalancer_dynamic_table_new_name [id=6ece274098c14d158c92bfa9d98f797c]) successfully deleted
I20240502 08:18:07.923151 27258 raft_consensus.cc:1261] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:07.923316 32747 ts_tablet_manager.cc:1875] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.923800 32646 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:07.933863 32448 raft_consensus.cc:1050] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6: attempting to promote NON_VOTER b01e7c8a302d41078fa39cfef665ff59 to VOTER
I20240502 08:18:07.935047 32448 consensus_queue.cc:237] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:07.939950 31848 raft_consensus.cc:1050] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1: attempting to promote NON_VOTER b01e7c8a302d41078fa39cfef665ff59 to VOTER
I20240502 08:18:07.940999 31848 consensus_queue.cc:237] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:07.948261 32717 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:07.952296 32747 ts_tablet_manager.cc:1888] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:07.955345 32747 log.cc:1196] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/33cc7f3146794b7f84c07e706fd26062
I20240502 08:18:07.956202 32747 ts_tablet_manager.cc:1909] T 33cc7f3146794b7f84c07e706fd26062 P 84cda20738d2452da166e31fc6da57a6: Deleting consensus metadata
I20240502 08:18:07.956688 30289 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:07.958133 27519 raft_consensus.cc:1261] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:07.958761 32680 consensus_queue.cc:1035] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:07.962801 27648 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:07.963611 27109 catalog_manager.cc:5484] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:07.966553 30027 raft_consensus.cc:1261] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:07.967226 30290 raft_consensus.cc:1261] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:07.967190 32680 consensus_queue.cc:1035] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:07.970196 32680 consensus_queue.cc:1035] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:07.976130 27116 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:07.993010 27238 tablet_service.cc:1467] Processing CreateTablet for tablet a564d51f718e4914896b91e61e271187 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=10595d62e5f14ab6aa6af14c25fc69a7]), partition=
I20240502 08:18:07.993321 30400 tablet_service.cc:1467] Processing CreateTablet for tablet a564d51f718e4914896b91e61e271187 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=10595d62e5f14ab6aa6af14c25fc69a7]), partition=
I20240502 08:18:07.993415 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a564d51f718e4914896b91e61e271187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:07.993695 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a564d51f718e4914896b91e61e271187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.022131 27519 raft_consensus.cc:1261] T 0d33916984034ab99f957026b3e30d38 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:08.022902 32763 tablet_bootstrap.cc:492] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:08.023175 32762 tablet_bootstrap.cc:492] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:08.024664 32763 tablet_bootstrap.cc:654] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.030844 32762 tablet_bootstrap.cc:654] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.031579 27257 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:08.033562 31848 raft_consensus.cc:2932] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:08.037686 32763 tablet_bootstrap.cc:492] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:08.037876 32763 ts_tablet_manager.cc:1367] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240502 08:18:08.038784 32763 raft_consensus.cc:348] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:08.039445 32763 raft_consensus.cc:374] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.039573 32763 raft_consensus.cc:729] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.039839 32763 consensus_queue.cc:260] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:08.040684 32763 ts_tablet_manager.cc:1398] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:08.043036 30289 raft_consensus.cc:2932] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:08.053156 27116 catalog_manager.cc:5484] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:08.054729 27109 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:08.070592 27370 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd5657c5cfd4ca4a78cdfdc503b91a8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=bb73dde083304fce8a7da7ac8fdd1d8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.070427 30025 raft_consensus.cc:2932] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:08.071148 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd5657c5cfd4ca4a78cdfdc503b91a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.076756 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd5657c5cfd4ca4a78cdfdc503b91a8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=bb73dde083304fce8a7da7ac8fdd1d8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.077263 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd5657c5cfd4ca4a78cdfdc503b91a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.086643 30400 tablet_service.cc:1467] Processing CreateTablet for tablet 67840e013729425e9ed2bb05dd6aa8aa (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=bb73dde083304fce8a7da7ac8fdd1d8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.087373 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67840e013729425e9ed2bb05dd6aa8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.096964 30007 tablet_service.cc:1467] Processing CreateTablet for tablet 67840e013729425e9ed2bb05dd6aa8aa (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=bb73dde083304fce8a7da7ac8fdd1d8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.097398 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67840e013729425e9ed2bb05dd6aa8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.097486 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 67840e013729425e9ed2bb05dd6aa8aa (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=bb73dde083304fce8a7da7ac8fdd1d8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.097851 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67840e013729425e9ed2bb05dd6aa8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.102699 27629 tablet_service.cc:1467] Processing CreateTablet for tablet 2022f2450a6a4374b21596d9b871dd75 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=bb73dde083304fce8a7da7ac8fdd1d8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.103227 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2022f2450a6a4374b21596d9b871dd75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.112247 32172 raft_consensus.cc:482] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:08.112429 32172 raft_consensus.cc:504] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:08.113413 32172 leader_election.cc:288] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:08.114156 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a564d51f718e4914896b91e61e271187" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
W20240502 08:18:08.114866 27191 leader_election.cc:341] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:18:08.115320 27194 leader_election.cc:341] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Not found: Tablet not found: a564d51f718e4914896b91e61e271187
I20240502 08:18:08.115447 27194 leader_election.cc:302] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:08.116044 32462 raft_consensus.cc:2726] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:07.999405 30138 tablet_service.cc:1467] Processing CreateTablet for tablet a564d51f718e4914896b91e61e271187 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=10595d62e5f14ab6aa6af14c25fc69a7]), partition=
I20240502 08:18:08.120486 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a564d51f718e4914896b91e61e271187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.122334 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 2022f2450a6a4374b21596d9b871dd75 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=bb73dde083304fce8a7da7ac8fdd1d8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.122735 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2022f2450a6a4374b21596d9b871dd75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.123231 305 tablet_bootstrap.cc:492] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:08.124776 305 tablet_bootstrap.cc:654] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.022723 32437 consensus_queue.cc:1035] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:08.090687 30140 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd5657c5cfd4ca4a78cdfdc503b91a8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=bb73dde083304fce8a7da7ac8fdd1d8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.126492 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd5657c5cfd4ca4a78cdfdc503b91a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.128854 30027 raft_consensus.cc:1261] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:08.129395 300 consensus_queue.cc:1035] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:08.130692 30288 raft_consensus.cc:1261] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:08.131227 300 consensus_queue.cc:1035] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:08.097728 30139 tablet_service.cc:1467] Processing CreateTablet for tablet 2022f2450a6a4374b21596d9b871dd75 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=bb73dde083304fce8a7da7ac8fdd1d8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.131942 30139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2022f2450a6a4374b21596d9b871dd75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.140869 309 tablet_bootstrap.cc:492] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:08.142925 309 tablet_bootstrap.cc:654] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.146829 300 raft_consensus.cc:2932] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:08.148597 309 tablet_bootstrap.cc:492] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:08.148730 309 ts_tablet_manager.cc:1367] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 08:18:08.149466 309 raft_consensus.cc:348] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:08.149649 309 raft_consensus.cc:374] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.149746 309 raft_consensus.cc:729] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.149956 309 consensus_queue.cc:260] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:08.150970 30028 raft_consensus.cc:2932] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:08.154529 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet 33cc7f3146794b7f84c07e706fd26062 (table rebalancer_dynamic_table_new_name [id=6ece274098c14d158c92bfa9d98f797c]) successfully deleted
I20240502 08:18:08.158084 309 ts_tablet_manager.cc:1398] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240502 08:18:08.161908 27112 catalog_manager.cc:5484] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:08.169659 304 tablet_bootstrap.cc:492] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:08.169595 27257 consensus_queue.cc:237] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:08.171803 30422 raft_consensus.cc:1261] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:08.172212 304 tablet_bootstrap.cc:654] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.172639 32462 consensus_queue.cc:1035] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:08.174562 27516 raft_consensus.cc:2932] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:08.176288 32462 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:08.180567 30422 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:08.184448 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 20a836dd442943d08a87d61afc785cd1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:08.188144 27110 catalog_manager.cc:5484] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:08.187732 30289 raft_consensus.cc:2932] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:08.193713 27499 tablet_service.cc:1514] Processing DeleteTablet for tablet 20a836dd442943d08a87d61afc785cd1 with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:08.202927 310 tablet_bootstrap.cc:492] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:08.205453 310 tablet_bootstrap.cc:654] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.213048 27519 raft_consensus.cc:2932] T 0d33916984034ab99f957026b3e30d38 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:08.176443 27648 raft_consensus.cc:1261] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:08.214526 32462 consensus_queue.cc:1035] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:08.216370 27648 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:08.217542 30159 consensus_queue.cc:237] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:08.227581 27516 raft_consensus.cc:1261] T 0d33916984034ab99f957026b3e30d38 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:08.228538 32448 consensus_queue.cc:1035] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:08.231233 32437 raft_consensus.cc:2932] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:08.213655 317 tablet_replica.cc:330] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:08.234712 30028 raft_consensus.cc:1261] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:08.235934 317 raft_consensus.cc:2229] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.236313 317 raft_consensus.cc:2258] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.236977 302 consensus_queue.cc:1035] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:08.237952 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0d33916984034ab99f957026b3e30d38 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:08.239099 27516 raft_consensus.cc:2932] T 0d33916984034ab99f957026b3e30d38 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:08.239511 30028 raft_consensus.cc:2932] T 0d33916984034ab99f957026b3e30d38 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:08.248626 27109 catalog_manager.cc:5484] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:08.251513 317 ts_tablet_manager.cc:1875] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:08.241248 303 tablet_bootstrap.cc:492] T 67840e013729425e9ed2bb05dd6aa8aa P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:08.252465 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d33916984034ab99f957026b3e30d38 with delete_type TABLET_DATA_TOMBSTONED (TS 49c5c8f70e494edd9fcda40cc0af6cc4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:08.253222 32727 tablet_replica.cc:330] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:08.253441 32727 raft_consensus.cc:2229] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.253664 32727 raft_consensus.cc:2258] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.254566 32727 ts_tablet_manager.cc:1875] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:08.257467 304 tablet_bootstrap.cc:492] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:08.257604 304 ts_tablet_manager.cc:1367] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.088s user 0.004s sys 0.000s
I20240502 08:18:08.258453 304 raft_consensus.cc:348] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:08.260978 32727 ts_tablet_manager.cc:1888] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:08.261191 32727 log.cc:1196] T 0d33916984034ab99f957026b3e30d38 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/0d33916984034ab99f957026b3e30d38
I20240502 08:18:08.262115 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet 0d33916984034ab99f957026b3e30d38 (table rebalancer_extra_table_2 [id=e14b3e831e834278ac631fa1cb84dced]) successfully deleted
I20240502 08:18:08.263435 305 tablet_bootstrap.cc:492] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:18:08.265492 32762 tablet_bootstrap.cc:492] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:08.265769 32762 ts_tablet_manager.cc:1367] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.243s user 0.004s sys 0.002s
I20240502 08:18:08.266971 305 ts_tablet_manager.cc:1367] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.144s user 0.004s sys 0.013s
I20240502 08:18:08.266860 32762 raft_consensus.cc:348] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:08.267083 310 tablet_bootstrap.cc:492] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:08.267218 310 ts_tablet_manager.cc:1367] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.002s
I20240502 08:18:08.267243 32762 raft_consensus.cc:374] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 08:18:08.267650 29961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.7:40343, user_credentials={real_user=slave}} blocked reactor thread for 93937.5us
I20240502 08:18:08.267998 310 raft_consensus.cc:348] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:08.268222 310 raft_consensus.cc:374] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.268335 310 raft_consensus.cc:729] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.268558 310 consensus_queue.cc:260] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:08.269150 310 ts_tablet_manager.cc:1398] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:08.270259 305 raft_consensus.cc:348] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:08.271939 310 tablet_bootstrap.cc:492] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:08.273551 310 tablet_bootstrap.cc:654] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.278235 303 tablet_bootstrap.cc:654] T 67840e013729425e9ed2bb05dd6aa8aa P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.278420 304 raft_consensus.cc:374] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.279963 304 raft_consensus.cc:729] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.280254 305 raft_consensus.cc:374] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.280534 304 consensus_queue.cc:260] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:08.280891 305 raft_consensus.cc:729] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.281267 305 consensus_queue.cc:260] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:08.284067 310 tablet_bootstrap.cc:492] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:08.284309 310 ts_tablet_manager.cc:1367] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20240502 08:18:08.285261 310 raft_consensus.cc:348] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:08.285526 310 raft_consensus.cc:374] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.285641 310 raft_consensus.cc:729] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.285920 310 consensus_queue.cc:260] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:08.286074 304 ts_tablet_manager.cc:1398] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.028s user 0.003s sys 0.000s
I20240502 08:18:08.286603 310 ts_tablet_manager.cc:1398] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:08.287277 310 tablet_bootstrap.cc:492] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:08.288285 305 ts_tablet_manager.cc:1398] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240502 08:18:08.286368 304 tablet_bootstrap.cc:492] T 2022f2450a6a4374b21596d9b871dd75 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:08.288820 310 tablet_bootstrap.cc:654] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.291241 32628 raft_consensus.cc:482] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:08.291460 32628 raft_consensus.cc:504] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:08.292229 32628 leader_election.cc:288] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:08.293675 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd5657c5cfd4ca4a78cdfdc503b91a8" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:08.293900 30159 raft_consensus.cc:2454] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:08.295435 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd5657c5cfd4ca4a78cdfdc503b91a8" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:18:08.295670 27388 raft_consensus.cc:2454] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:08.296273 30225 leader_election.cc:302] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:08.296737 32628 raft_consensus.cc:2781] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:08.296841 32628 raft_consensus.cc:482] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:08.296929 32628 raft_consensus.cc:3037] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.300554 32628 raft_consensus.cc:504] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:08.300954 304 tablet_bootstrap.cc:654] T 2022f2450a6a4374b21596d9b871dd75 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.301627 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd5657c5cfd4ca4a78cdfdc503b91a8" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:08.301851 30159 raft_consensus.cc:3037] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.304519 30159 raft_consensus.cc:2454] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:08.305119 30226 leader_election.cc:302] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:08.305383 32044 raft_consensus.cc:2781] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:08.306913 303 tablet_bootstrap.cc:492] T 67840e013729425e9ed2bb05dd6aa8aa P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:08.307121 303 ts_tablet_manager.cc:1367] T 67840e013729425e9ed2bb05dd6aa8aa P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.000s
I20240502 08:18:08.307934 303 raft_consensus.cc:348] T 67840e013729425e9ed2bb05dd6aa8aa P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:08.308277 303 raft_consensus.cc:374] T 67840e013729425e9ed2bb05dd6aa8aa P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.308424 303 raft_consensus.cc:729] T 67840e013729425e9ed2bb05dd6aa8aa P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.308717 303 consensus_queue.cc:260] T 67840e013729425e9ed2bb05dd6aa8aa P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:08.309509 303 ts_tablet_manager.cc:1398] T 67840e013729425e9ed2bb05dd6aa8aa P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:08.315966 32628 leader_election.cc:288] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:08.319649 317 ts_tablet_manager.cc:1888] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:08.320531 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd5657c5cfd4ca4a78cdfdc503b91a8" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:18:08.320775 27388 raft_consensus.cc:3037] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.322847 32763 tablet_bootstrap.cc:492] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:08.322854 32762 raft_consensus.cc:729] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.323148 32762 consensus_queue.cc:260] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:08.323740 32762 ts_tablet_manager.cc:1398] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.058s user 0.000s sys 0.001s
I20240502 08:18:08.324247 32763 tablet_bootstrap.cc:654] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.325215 27388 raft_consensus.cc:2454] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:08.334769 32437 raft_consensus.cc:482] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:08.334980 32437 raft_consensus.cc:504] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:08.335747 32437 leader_election.cc:288] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:08.337939 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2022f2450a6a4374b21596d9b871dd75" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
W20240502 08:18:08.340018 30095 leader_election.cc:341] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:08.341150 27648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2022f2450a6a4374b21596d9b871dd75" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:08.342742 27388 consensus_queue.cc:237] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:08.345031 32763 tablet_bootstrap.cc:492] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:08.345243 32763 ts_tablet_manager.cc:1367] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.004s
W20240502 08:18:08.345611 30096 leader_election.cc:341] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:08.345786 30096 leader_election.cc:302] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442
I20240502 08:18:08.345938 32763 raft_consensus.cc:348] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:08.346180 32763 raft_consensus.cc:374] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.346242 32763 raft_consensus.cc:729] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.346462 32763 consensus_queue.cc:260] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:08.348414 317 log.cc:1196] T 20a836dd442943d08a87d61afc785cd1 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/20a836dd442943d08a87d61afc785cd1
I20240502 08:18:08.348599 32448 raft_consensus.cc:2726] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:08.349261 32762 tablet_bootstrap.cc:492] T 67840e013729425e9ed2bb05dd6aa8aa P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:08.349931 32763 ts_tablet_manager.cc:1398] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 08:18:08.349928 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 20a836dd442943d08a87d61afc785cd1 (table rebalance_test_table_19 [id=7f9af7f2cb49474eb721698045f06a63]) successfully deleted
I20240502 08:18:08.351452 32762 tablet_bootstrap.cc:654] T 67840e013729425e9ed2bb05dd6aa8aa P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.352232 27516 raft_consensus.cc:1261] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:08.352819 31848 consensus_queue.cc:1035] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:08.355424 31848 raft_consensus.cc:2932] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:08.356156 30289 raft_consensus.cc:1261] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:08.356758 313 consensus_queue.cc:1035] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:08.358045 329 raft_consensus.cc:482] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:08.358249 329 raft_consensus.cc:504] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:08.358390 30288 raft_consensus.cc:2932] T f63eec7e63fe4e71b1958903e0d2cb00 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:08.359082 329 leader_election.cc:288] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:08.359125 27516 raft_consensus.cc:2932] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:08.360422 310 tablet_bootstrap.cc:492] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:08.360603 310 ts_tablet_manager.cc:1367] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.074s user 0.006s sys 0.008s
I20240502 08:18:08.361375 310 raft_consensus.cc:348] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:08.361603 310 raft_consensus.cc:374] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.361724 310 raft_consensus.cc:729] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.362021 310 consensus_queue.cc:260] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:08.362735 310 ts_tablet_manager.cc:1398] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:08.365526 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f63eec7e63fe4e71b1958903e0d2cb00 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:08.369776 32044 raft_consensus.cc:686] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:18:08.369807 27110 catalog_manager.cc:5484] T f63eec7e63fe4e71b1958903e0d2cb00 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:08.370138 32044 consensus_queue.cc:237] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:08.372022 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67840e013729425e9ed2bb05dd6aa8aa" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:08.372277 30027 raft_consensus.cc:2454] T 67840e013729425e9ed2bb05dd6aa8aa P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:18:08.372692 27584 leader_election.cc:302] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:08.373028 329 raft_consensus.cc:2781] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:08.373198 329 raft_consensus.cc:482] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:08.373345 329 raft_consensus.cc:3037] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.374094 30007 tablet_service.cc:1514] Processing DeleteTablet for tablet f63eec7e63fe4e71b1958903e0d2cb00 with delete_type TABLET_DATA_TOMBSTONED (TS 1f0bf4bc3c5146539146b825b048bff3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:08.361863 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67840e013729425e9ed2bb05dd6aa8aa" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
W20240502 08:18:08.381116 27584 leader_election.cc:341] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:08.384438 27110 catalog_manager.cc:5484] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7). New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:08.381950 329 raft_consensus.cc:504] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:08.396966 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67840e013729425e9ed2bb05dd6aa8aa" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
W20240502 08:18:08.397730 27584 leader_election.cc:341] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:08.398820 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67840e013729425e9ed2bb05dd6aa8aa" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:08.399183 30028 raft_consensus.cc:3037] T 67840e013729425e9ed2bb05dd6aa8aa P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.399514 329 leader_election.cc:288] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:08.403282 331 tablet_replica.cc:330] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:08.403553 32762 tablet_bootstrap.cc:492] T 67840e013729425e9ed2bb05dd6aa8aa P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:08.403653 331 raft_consensus.cc:2229] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.403831 32762 ts_tablet_manager.cc:1367] T 67840e013729425e9ed2bb05dd6aa8aa P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.001s
I20240502 08:18:08.403935 331 raft_consensus.cc:2258] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.404611 32762 raft_consensus.cc:348] T 67840e013729425e9ed2bb05dd6aa8aa P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:08.404913 32762 raft_consensus.cc:374] T 67840e013729425e9ed2bb05dd6aa8aa P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.405011 32762 raft_consensus.cc:729] T 67840e013729425e9ed2bb05dd6aa8aa P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.405263 32762 consensus_queue.cc:260] T 67840e013729425e9ed2bb05dd6aa8aa P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:08.405798 32762 ts_tablet_manager.cc:1398] T 67840e013729425e9ed2bb05dd6aa8aa P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:08.405896 30028 raft_consensus.cc:2454] T 67840e013729425e9ed2bb05dd6aa8aa P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:08.406296 27584 leader_election.cc:302] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, bf8ac71b15f740a0a928911360773442; no voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad
I20240502 08:18:08.406684 329 raft_consensus.cc:2781] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:08.406889 329 raft_consensus.cc:686] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:18:08.407184 329 consensus_queue.cc:237] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:08.408386 331 ts_tablet_manager.cc:1875] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:08.410446 328 raft_consensus.cc:482] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:08.411955 328 raft_consensus.cc:504] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:08.413228 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a564d51f718e4914896b91e61e271187" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:08.413419 27257 raft_consensus.cc:2454] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:08.413830 30357 leader_election.cc:302] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters:
I20240502 08:18:08.414110 32647 raft_consensus.cc:2781] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:08.414240 32647 raft_consensus.cc:482] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:08.414347 32647 raft_consensus.cc:3037] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.414462 328 leader_election.cc:288] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:08.414690 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a564d51f718e4914896b91e61e271187" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:08.414911 30159 raft_consensus.cc:2454] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:08.417227 32647 raft_consensus.cc:504] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:08.418313 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a564d51f718e4914896b91e61e271187" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:08.418560 32647 leader_election.cc:288] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:08.418570 27257 raft_consensus.cc:3037] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.419916 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a564d51f718e4914896b91e61e271187" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:08.420111 30159 raft_consensus.cc:3037] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.423501 27112 catalog_manager.cc:5484] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:08.424741 30159 raft_consensus.cc:2454] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:08.425127 30358 leader_election.cc:302] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:08.425529 32647 raft_consensus.cc:2781] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:08.425696 32647 raft_consensus.cc:686] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:18:08.425951 32647 consensus_queue.cc:237] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:08.428905 27257 raft_consensus.cc:2454] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:08.439425 328 raft_consensus.cc:482] T 67840e013729425e9ed2bb05dd6aa8aa P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:08.439603 328 raft_consensus.cc:504] T 67840e013729425e9ed2bb05dd6aa8aa P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:08.440361 328 leader_election.cc:288] T 67840e013729425e9ed2bb05dd6aa8aa P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:08.441267 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67840e013729425e9ed2bb05dd6aa8aa" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:08.445102 304 tablet_bootstrap.cc:492] T 2022f2450a6a4374b21596d9b871dd75 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:08.446646 304 ts_tablet_manager.cc:1367] T 2022f2450a6a4374b21596d9b871dd75 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.160s user 0.020s sys 0.015s
I20240502 08:18:08.449167 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67840e013729425e9ed2bb05dd6aa8aa" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:08.449506 30028 raft_consensus.cc:2379] T 67840e013729425e9ed2bb05dd6aa8aa P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in current term 1: Already voted for candidate bf8ac71b15f740a0a928911360773442 in this term.
I20240502 08:18:08.449919 30355 leader_election.cc:302] T 67840e013729425e9ed2bb05dd6aa8aa P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 1f0bf4bc3c5146539146b825b048bff3, bf8ac71b15f740a0a928911360773442
I20240502 08:18:08.450338 328 raft_consensus.cc:3037] T 67840e013729425e9ed2bb05dd6aa8aa P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.450999 304 raft_consensus.cc:348] T 2022f2450a6a4374b21596d9b871dd75 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:08.451197 304 raft_consensus.cc:374] T 2022f2450a6a4374b21596d9b871dd75 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.451259 304 raft_consensus.cc:729] T 2022f2450a6a4374b21596d9b871dd75 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.452548 304 consensus_queue.cc:260] T 2022f2450a6a4374b21596d9b871dd75 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:08.453189 328 raft_consensus.cc:2726] T 67840e013729425e9ed2bb05dd6aa8aa P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:08.456811 27110 catalog_manager.cc:5484] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:08.459721 304 ts_tablet_manager.cc:1398] T 2022f2450a6a4374b21596d9b871dd75 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240502 08:18:08.466971 331 ts_tablet_manager.cc:1888] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:08.467175 331 log.cc:1196] T f63eec7e63fe4e71b1958903e0d2cb00 P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/f63eec7e63fe4e71b1958903e0d2cb00
I20240502 08:18:08.467985 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet f63eec7e63fe4e71b1958903e0d2cb00 (table rebalancer_extra_table_6 [id=e57c96062aca4609bc2d923cb7a30a29]) successfully deleted
I20240502 08:18:08.468533 32732 tablet_bootstrap.cc:492] T a3724af3b82c43b0a1036f4d9703a3a9 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:08.469000 32732 tablet_bootstrap.cc:492] T a3724af3b82c43b0a1036f4d9703a3a9 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap complete.
I20240502 08:18:08.469411 32732 ts_tablet_manager.cc:1367] T a3724af3b82c43b0a1036f4d9703a3a9 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.626s user 0.060s sys 0.009s
I20240502 08:18:08.470237 32732 raft_consensus.cc:348] T a3724af3b82c43b0a1036f4d9703a3a9 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:08.470657 32732 raft_consensus.cc:729] T a3724af3b82c43b0a1036f4d9703a3a9 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: LEARNER
I20240502 08:18:08.470863 32732 consensus_queue.cc:260] T a3724af3b82c43b0a1036f4d9703a3a9 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:08.476203 32732 ts_tablet_manager.cc:1398] T a3724af3b82c43b0a1036f4d9703a3a9 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 08:18:08.481509 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session 1f0bf4bc3c5146539146b825b048bff3-a3724af3b82c43b0a1036f4d9703a3a9 received from {username='slave'} at 127.20.207.5:51633
I20240502 08:18:08.481621 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session 1f0bf4bc3c5146539146b825b048bff3-a3724af3b82c43b0a1036f4d9703a3a9 on tablet a3724af3b82c43b0a1036f4d9703a3a9 with peer 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:18:08.497325 27516 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "903502e57ab2430383807b0d46a39847"
dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60712
I20240502 08:18:08.497586 27516 raft_consensus.cc:595] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:18:08.536185 30159 consensus_queue.cc:237] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:08.538751 30289 raft_consensus.cc:1261] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:08.538942 30422 raft_consensus.cc:1261] T 0a5a0175799e478c9f53ecc82ba090ae P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:08.539322 32448 consensus_queue.cc:1035] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:08.539671 32437 consensus_queue.cc:1035] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:08.540242 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a5a0175799e478c9f53ecc82ba090ae. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:08.541857 32448 raft_consensus.cc:2932] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:08.544013 30421 raft_consensus.cc:2932] T 0a5a0175799e478c9f53ecc82ba090ae P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:08.551123 27110 catalog_manager.cc:5484] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:08.549229 30289 raft_consensus.cc:2932] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:08.571166 27116 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:08.578126 30159 raft_consensus.cc:1261] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:08.578352 27257 raft_consensus.cc:1261] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:08.578944 328 consensus_queue.cc:1035] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.579416 328 consensus_queue.cc:1035] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.597223 27116 catalog_manager.cc:5576] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899) does not have the latest schema for tablet a564d51f718e4914896b91e61e271187 (table rebalancer_dynamic_table [id=10595d62e5f14ab6aa6af14c25fc69a7]). Expected version 1 got 0
I20240502 08:18:08.597772 27110 catalog_manager.cc:5576] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505) does not have the latest schema for tablet a564d51f718e4914896b91e61e271187 (table rebalancer_dynamic_table [id=10595d62e5f14ab6aa6af14c25fc69a7]). Expected version 1 got 0
I20240502 08:18:08.602115 385 tablet.cc:1722] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:08.603011 385 tablet.cc:1620] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:08.604907 388 tablet.cc:1722] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:08.608644 27110 catalog_manager.cc:5576] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899) does not have the latest schema for tablet a564d51f718e4914896b91e61e271187 (table rebalancer_dynamic_table [id=10595d62e5f14ab6aa6af14c25fc69a7]). Expected version 1 got 0
I20240502 08:18:08.610802 388 tablet.cc:1620] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:18:08.612596 27110 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet a564d51f718e4914896b91e61e271187 (table rebalancer_dynamic_table [id=10595d62e5f14ab6aa6af14c25fc69a7]). Expected version 1 got 0
I20240502 08:18:08.614921 32646 raft_consensus.cc:979] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59: : Instructing follower 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 to start an election
I20240502 08:18:08.615097 32646 raft_consensus.cc:1067] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Signalling peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 to start an election
I20240502 08:18:08.621258 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=10595d62e5f14ab6aa6af14c25fc69a7] alter complete (version 1)
I20240502 08:18:08.623376 27257 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "903502e57ab2430383807b0d46a39847"
dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
from {username='slave'} at 127.20.207.3:51699
I20240502 08:18:08.623600 27257 raft_consensus.cc:482] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:08.623687 27257 raft_consensus.cc:3037] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:08.626555 27257 raft_consensus.cc:504] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:08.627846 27257 leader_election.cc:288] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:08.629382 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903502e57ab2430383807b0d46a39847" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:18:08.629601 27516 raft_consensus.cc:3032] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:08.629678 27516 raft_consensus.cc:729] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:18:08.629681 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903502e57ab2430383807b0d46a39847" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:08.629913 27650 raft_consensus.cc:3037] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:08.630025 27516 consensus_queue.cc:260] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:08.630177 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903502e57ab2430383807b0d46a39847" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:08.630357 30289 raft_consensus.cc:3037] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:08.630474 27516 raft_consensus.cc:3037] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:08.632824 27650 raft_consensus.cc:2454] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 2.
I20240502 08:18:08.633435 30289 raft_consensus.cc:2454] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 2.
I20240502 08:18:08.633852 27516 raft_consensus.cc:2454] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 2.
I20240502 08:18:08.633852 27191 leader_election.cc:302] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 49c5c8f70e494edd9fcda40cc0af6cc4, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:08.634181 32462 raft_consensus.cc:2781] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:08.634316 32462 raft_consensus.cc:686] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:18:08.634562 32462 consensus_queue.cc:237] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:08.639622 27112 catalog_manager.cc:5484] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 1 to 2, leader changed from b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 2 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:08.655565 30028 raft_consensus.cc:1203] T a3724af3b82c43b0a1036f4d9703a3a9 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:08.674312 400 tablet.cc:1722] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:08.675683 400 tablet.cc:1620] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: MemRowSet was empty: no flush needed.
I20240502 08:18:08.677186 27112 catalog_manager.cc:5576] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505) does not have the latest schema for tablet a564d51f718e4914896b91e61e271187 (table rebalancer_dynamic_table [id=10595d62e5f14ab6aa6af14c25fc69a7]). Expected version 1 got 0
I20240502 08:18:08.694242 27519 consensus_queue.cc:237] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:08.696282 27257 raft_consensus.cc:1261] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:08.696406 27650 raft_consensus.cc:1261] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:08.696954 32721 consensus_queue.cc:1035] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:08.697247 32646 consensus_queue.cc:1035] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:18:08.698280 27453 consensus_peers.cc:485] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 -> Peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Couldn't send request to peer 1f0bf4bc3c5146539146b825b048bff3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9725e08a6c5d4410b8d944ab154ec1eb. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:08.700152 32646 raft_consensus.cc:2932] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:08.702697 27257 raft_consensus.cc:2932] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:08.706370 27116 catalog_manager.cc:5484] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:08.723630 27648 raft_consensus.cc:2932] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:08.725663 27112 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:08.735445 385 tablet.cc:1722] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:08.735922 385 tablet.cc:1620] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:08.736979 388 tablet.cc:1722] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:08.737560 388 tablet.cc:1620] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:18:08.751387 27112 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=10595d62e5f14ab6aa6af14c25fc69a7] alter complete (version 2)
I20240502 08:18:08.753667 400 tablet.cc:1722] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:08.754125 400 tablet.cc:1620] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: MemRowSet was empty: no flush needed.
I20240502 08:18:08.766264 392 raft_consensus.cc:482] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:08.766408 392 raft_consensus.cc:504] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:08.767043 392 leader_election.cc:288] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:08.767716 27648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2022f2450a6a4374b21596d9b871dd75" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:08.767932 27648 raft_consensus.cc:2454] T 2022f2450a6a4374b21596d9b871dd75 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:18:08.768352 27194 leader_election.cc:302] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:08.768880 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2022f2450a6a4374b21596d9b871dd75" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:08.769102 30159 raft_consensus.cc:2454] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:18:08.769668 392 raft_consensus.cc:2781] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:08.769742 392 raft_consensus.cc:482] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:08.769793 392 raft_consensus.cc:3037] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.772574 392 raft_consensus.cc:504] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:08.773537 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2022f2450a6a4374b21596d9b871dd75" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:08.773726 30159 raft_consensus.cc:3037] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.774354 392 leader_election.cc:288] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:08.776521 30159 raft_consensus.cc:2454] T 2022f2450a6a4374b21596d9b871dd75 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:18:08.777024 27194 leader_election.cc:302] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:08.777971 32172 raft_consensus.cc:2781] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:08.778122 32172 raft_consensus.cc:686] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:18:08.778352 32172 consensus_queue.cc:237] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:08.780527 27648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2022f2450a6a4374b21596d9b871dd75" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:08.780787 27648 raft_consensus.cc:3037] T 2022f2450a6a4374b21596d9b871dd75 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.783720 27648 raft_consensus.cc:2454] T 2022f2450a6a4374b21596d9b871dd75 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:18:08.790870 27112 catalog_manager.cc:5484] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:08.796779 324 consensus_queue.cc:1035] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.833460 329 consensus_queue.cc:1035] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.847579 32628 consensus_queue.cc:1035] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.855355 27116 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:08.857620 31753 consensus_queue.cc:1035] T 67840e013729425e9ed2bb05dd6aa8aa P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.865947 385 tablet.cc:1722] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:08.866442 388 tablet.cc:1722] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:08.870046 400 tablet.cc:1722] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:08.878261 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=10595d62e5f14ab6aa6af14c25fc69a7] alter complete (version 3)
I20240502 08:18:08.981827 27112 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:08.982076 27112 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:08.985397 27112 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet a564d51f718e4914896b91e61e271187
I20240502 08:18:08.986140 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet a564d51f718e4914896b91e61e271187 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:18:08.986202 30140 tablet_service.cc:1514] Processing DeleteTablet for tablet a564d51f718e4914896b91e61e271187 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:08.986770 30400 tablet_service.cc:1514] Processing DeleteTablet for tablet a564d51f718e4914896b91e61e271187 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:08.989145 440 tablet_replica.cc:330] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:08.989367 440 raft_consensus.cc:2229] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.989605 440 raft_consensus.cc:2258] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.992060 440 ts_tablet_manager.cc:1875] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.996384 440 ts_tablet_manager.cc:1888] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:08.996529 440 log.cc:1196] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/a564d51f718e4914896b91e61e271187
I20240502 08:18:08.996904 440 ts_tablet_manager.cc:1909] T a564d51f718e4914896b91e61e271187 P 84cda20738d2452da166e31fc6da57a6: Deleting consensus metadata
I20240502 08:18:08.997597 439 tablet_replica.cc:330] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:18:08.997769 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet a564d51f718e4914896b91e61e271187 (table rebalancer_dynamic_table_new_name [id=10595d62e5f14ab6aa6af14c25fc69a7]) successfully deleted
I20240502 08:18:08.997807 439 raft_consensus.cc:2229] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.998054 439 raft_consensus.cc:2258] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.999085 439 ts_tablet_manager.cc:1875] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.999424 441 tablet_replica.cc:330] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:08.999619 441 raft_consensus.cc:2229] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:08.999944 441 raft_consensus.cc:2258] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:09.000591 441 ts_tablet_manager.cc:1875] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:09.005033 439 ts_tablet_manager.cc:1888] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:09.005141 439 log.cc:1196] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/a564d51f718e4914896b91e61e271187
I20240502 08:18:09.005524 439 ts_tablet_manager.cc:1909] T a564d51f718e4914896b91e61e271187 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:18:09.006484 441 ts_tablet_manager.cc:1888] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:09.006515 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet a564d51f718e4914896b91e61e271187 (table rebalancer_dynamic_table_new_name [id=10595d62e5f14ab6aa6af14c25fc69a7]) successfully deleted
I20240502 08:18:09.006585 441 log.cc:1196] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/a564d51f718e4914896b91e61e271187
I20240502 08:18:09.006927 441 ts_tablet_manager.cc:1909] T a564d51f718e4914896b91e61e271187 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting consensus metadata
I20240502 08:18:09.007680 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet a564d51f718e4914896b91e61e271187 (table rebalancer_dynamic_table_new_name [id=10595d62e5f14ab6aa6af14c25fc69a7]) successfully deleted
I20240502 08:18:09.043195 32172 raft_consensus.cc:1050] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 to VOTER
I20240502 08:18:09.043998 32172 consensus_queue.cc:237] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:09.046139 27388 raft_consensus.cc:1261] T a3724af3b82c43b0a1036f4d9703a3a9 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:09.046396 30027 raft_consensus.cc:1261] T a3724af3b82c43b0a1036f4d9703a3a9 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:09.046525 27519 raft_consensus.cc:1261] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:09.046631 32172 consensus_queue.cc:1035] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:09.046854 392 consensus_queue.cc:1035] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:09.047155 32462 consensus_queue.cc:1035] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:09.050410 392 raft_consensus.cc:2932] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:09.050930 30027 raft_consensus.cc:2932] T a3724af3b82c43b0a1036f4d9703a3a9 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:09.051108 27516 raft_consensus.cc:2932] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:09.054555 27388 raft_consensus.cc:2932] T a3724af3b82c43b0a1036f4d9703a3a9 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:09.056640 27112 catalog_manager.cc:5484] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:09.064555 27258 consensus_queue.cc:237] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:09.066340 30027 raft_consensus.cc:1261] T a3724af3b82c43b0a1036f4d9703a3a9 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:09.066829 32172 consensus_queue.cc:1035] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:09.067010 27388 raft_consensus.cc:1261] T a3724af3b82c43b0a1036f4d9703a3a9 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:09.067589 32462 consensus_queue.cc:1035] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:09.070482 32462 raft_consensus.cc:2932] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:09.071823 30027 raft_consensus.cc:2932] T a3724af3b82c43b0a1036f4d9703a3a9 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:09.074056 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a3724af3b82c43b0a1036f4d9703a3a9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:09.074795 27388 raft_consensus.cc:2932] T a3724af3b82c43b0a1036f4d9703a3a9 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:09.077586 27110 catalog_manager.cc:5484] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 5 to 6, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:09.083782 27499 tablet_service.cc:1514] Processing DeleteTablet for tablet a3724af3b82c43b0a1036f4d9703a3a9 with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:09.084343 446 tablet_replica.cc:330] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:09.084543 446 raft_consensus.cc:2229] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:09.084723 446 raft_consensus.cc:2258] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:09.085741 446 ts_tablet_manager.cc:1875] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:09.090085 27111 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:09.090788 446 ts_tablet_manager.cc:1888] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:09.091001 446 log.cc:1196] T a3724af3b82c43b0a1036f4d9703a3a9 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/a3724af3b82c43b0a1036f4d9703a3a9
I20240502 08:18:09.091890 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet a3724af3b82c43b0a1036f4d9703a3a9 (table rebalance_test_table_17 [id=c179f8e244074d9ead49cbf91748c01e]) successfully deleted
I20240502 08:18:09.099377 30140 tablet_service.cc:1467] Processing CreateTablet for tablet 125f21b7543143f5ba6c52a1385c37d5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3197a001f13546fbb549cdb7db5750c2]), partition=
I20240502 08:18:09.099756 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 125f21b7543143f5ba6c52a1385c37d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.100914 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 125f21b7543143f5ba6c52a1385c37d5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3197a001f13546fbb549cdb7db5750c2]), partition=
I20240502 08:18:09.101290 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 125f21b7543143f5ba6c52a1385c37d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.104638 30007 tablet_service.cc:1467] Processing CreateTablet for tablet 125f21b7543143f5ba6c52a1385c37d5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3197a001f13546fbb549cdb7db5750c2]), partition=
I20240502 08:18:09.105080 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 125f21b7543143f5ba6c52a1385c37d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.117516 450 ts_tablet_manager.cc:902] T 0a5a0175799e478c9f53ecc82ba090ae P b01e7c8a302d41078fa39cfef665ff59: Initiating tablet copy from peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:09.119976 450 tablet_copy_client.cc:308] T 0a5a0175799e478c9f53ecc82ba090ae P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.6:38899
I20240502 08:18:09.121008 27516 raft_consensus.cc:1261] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 2 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:18:09.121268 30180 tablet_copy_service.cc:140] P 84cda20738d2452da166e31fc6da57a6: Received BeginTabletCopySession request for tablet 0a5a0175799e478c9f53ecc82ba090ae from peer b01e7c8a302d41078fa39cfef665ff59 ({username='slave'} at 127.20.207.3:35877)
I20240502 08:18:09.121399 30180 tablet_copy_service.cc:161] P 84cda20738d2452da166e31fc6da57a6: Beginning new tablet copy session on tablet 0a5a0175799e478c9f53ecc82ba090ae from peer b01e7c8a302d41078fa39cfef665ff59 at {username='slave'} at 127.20.207.3:35877: session id = b01e7c8a302d41078fa39cfef665ff59-0a5a0175799e478c9f53ecc82ba090ae
I20240502 08:18:09.121485 32172 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:09.125519 30180 tablet_copy_source_session.cc:205] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:09.126915 450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5a0175799e478c9f53ecc82ba090ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.132109 450 tablet_copy_client.cc:783] T 0a5a0175799e478c9f53ecc82ba090ae P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:09.132285 450 tablet_copy_client.cc:647] T 0a5a0175799e478c9f53ecc82ba090ae P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:09.132356 457 tablet_bootstrap.cc:492] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:09.134550 450 tablet_copy_client.cc:523] T 0a5a0175799e478c9f53ecc82ba090ae P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:09.135533 457 tablet_bootstrap.cc:654] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.137241 456 tablet_bootstrap.cc:492] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:09.137496 450 tablet_bootstrap.cc:492] T 0a5a0175799e478c9f53ecc82ba090ae P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:09.140292 456 tablet_bootstrap.cc:654] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.144264 450 tablet_bootstrap.cc:492] T 0a5a0175799e478c9f53ecc82ba090ae P b01e7c8a302d41078fa39cfef665ff59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:09.144498 450 tablet_bootstrap.cc:492] T 0a5a0175799e478c9f53ecc82ba090ae P b01e7c8a302d41078fa39cfef665ff59: Bootstrap complete.
I20240502 08:18:09.144661 450 ts_tablet_manager.cc:1367] T 0a5a0175799e478c9f53ecc82ba090ae P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 08:18:09.144883 449 tablet_bootstrap.cc:492] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:09.145275 450 raft_consensus.cc:348] T 0a5a0175799e478c9f53ecc82ba090ae P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:09.145432 450 raft_consensus.cc:729] T 0a5a0175799e478c9f53ecc82ba090ae P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: LEARNER
I20240502 08:18:09.145558 450 consensus_queue.cc:260] T 0a5a0175799e478c9f53ecc82ba090ae P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:09.145668 457 tablet_bootstrap.cc:492] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:09.145984 457 ts_tablet_manager.cc:1367] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240502 08:18:09.146023 450 ts_tablet_manager.cc:1398] T 0a5a0175799e478c9f53ecc82ba090ae P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:18:09.147953 30180 tablet_copy_service.cc:342] P 84cda20738d2452da166e31fc6da57a6: Request end of tablet copy session b01e7c8a302d41078fa39cfef665ff59-0a5a0175799e478c9f53ecc82ba090ae received from {username='slave'} at 127.20.207.3:35877
I20240502 08:18:09.148139 30180 tablet_copy_service.cc:433] P 84cda20738d2452da166e31fc6da57a6: ending tablet copy session b01e7c8a302d41078fa39cfef665ff59-0a5a0175799e478c9f53ecc82ba090ae on tablet 0a5a0175799e478c9f53ecc82ba090ae with peer b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:09.149839 457 raft_consensus.cc:348] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:09.150178 457 raft_consensus.cc:374] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.150775 449 tablet_bootstrap.cc:654] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.150319 457 raft_consensus.cc:729] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.150991 27648 raft_consensus.cc:1261] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:18:09.151450 32172 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:09.151369 457 consensus_queue.cc:260] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:09.154932 457 ts_tablet_manager.cc:1398] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 08:18:09.161193 30289 raft_consensus.cc:1261] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:18:09.162503 32175 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:09.167021 449 tablet_bootstrap.cc:492] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:09.167207 449 ts_tablet_manager.cc:1367] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.000s
I20240502 08:18:09.168023 449 raft_consensus.cc:348] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:09.169677 449 raft_consensus.cc:374] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.169824 449 raft_consensus.cc:729] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.170079 449 consensus_queue.cc:260] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:09.170908 456 tablet_bootstrap.cc:492] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:09.171054 456 ts_tablet_manager.cc:1367] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20240502 08:18:09.171828 456 raft_consensus.cc:348] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:09.172032 456 raft_consensus.cc:374] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.172106 456 raft_consensus.cc:729] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.172318 456 consensus_queue.cc:260] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:09.172739 456 ts_tablet_manager.cc:1398] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:09.174139 32175 consensus_queue.cc:1035] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:09.176115 449 ts_tablet_manager.cc:1398] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 08:18:09.189036 32172 consensus_queue.cc:1035] T 2022f2450a6a4374b21596d9b871dd75 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:09.295627 501 ts_tablet_manager.cc:902] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:09.297295 501 tablet_copy_client.cc:308] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:09.297868 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet 9725e08a6c5d4410b8d944ab154ec1eb from peer 1f0bf4bc3c5146539146b825b048bff3 ({username='slave'} at 127.20.207.5:54501)
I20240502 08:18:09.298040 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet 9725e08a6c5d4410b8d944ab154ec1eb from peer 1f0bf4bc3c5146539146b825b048bff3 at {username='slave'} at 127.20.207.5:54501: session id = 1f0bf4bc3c5146539146b825b048bff3-9725e08a6c5d4410b8d944ab154ec1eb
I20240502 08:18:09.299788 27541 tablet_copy_source_session.cc:205] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:09.300884 501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9725e08a6c5d4410b8d944ab154ec1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.305853 501 tablet_copy_client.cc:783] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:09.306151 501 tablet_copy_client.cc:647] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:09.307646 501 tablet_copy_client.cc:523] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:09.309032 314 raft_consensus.cc:482] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:09.309171 314 raft_consensus.cc:504] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:09.310107 314 leader_election.cc:288] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:09.310364 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125f21b7543143f5ba6c52a1385c37d5" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:09.310598 30289 raft_consensus.cc:2454] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:09.310688 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125f21b7543143f5ba6c52a1385c37d5" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:09.310961 30159 raft_consensus.cc:2454] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:09.311295 501 tablet_bootstrap.cc:492] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:09.311344 29964 leader_election.cc:302] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:09.311756 314 raft_consensus.cc:2781] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:09.311918 314 raft_consensus.cc:482] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:09.312031 314 raft_consensus.cc:3037] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.315081 314 raft_consensus.cc:504] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:09.316031 314 leader_election.cc:288] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:09.316109 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125f21b7543143f5ba6c52a1385c37d5" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:09.316133 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125f21b7543143f5ba6c52a1385c37d5" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:09.316298 30289 raft_consensus.cc:3037] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.316321 30159 raft_consensus.cc:3037] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.318499 30289 raft_consensus.cc:2454] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:09.318871 30159 raft_consensus.cc:2454] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:09.319310 29961 leader_election.cc:302] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:18:09.319577 314 raft_consensus.cc:2781] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:09.322208 314 raft_consensus.cc:686] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:18:09.322544 314 consensus_queue.cc:237] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:09.329452 27111 catalog_manager.cc:5484] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:09.352507 501 tablet_bootstrap.cc:492] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:09.352895 501 tablet_bootstrap.cc:492] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap complete.
I20240502 08:18:09.353132 501 ts_tablet_manager.cc:1367] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.042s user 0.027s sys 0.006s
I20240502 08:18:09.353755 501 raft_consensus.cc:348] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:09.353967 501 raft_consensus.cc:729] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: LEARNER
I20240502 08:18:09.354193 501 consensus_queue.cc:260] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:09.354710 501 ts_tablet_manager.cc:1398] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 08:18:09.356088 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 1f0bf4bc3c5146539146b825b048bff3-9725e08a6c5d4410b8d944ab154ec1eb received from {username='slave'} at 127.20.207.5:54501
I20240502 08:18:09.356187 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 1f0bf4bc3c5146539146b825b048bff3-9725e08a6c5d4410b8d944ab154ec1eb on tablet 9725e08a6c5d4410b8d944ab154ec1eb with peer 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:18:09.423556 27111 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:09.431452 27238 tablet_service.cc:1467] Processing CreateTablet for tablet e5af418a0ae043a387b5030a00b47e91 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8fad8146798f48ce9901b24b7f1944b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.431758 30400 tablet_service.cc:1467] Processing CreateTablet for tablet e5af418a0ae043a387b5030a00b47e91 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8fad8146798f48ce9901b24b7f1944b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.431895 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5af418a0ae043a387b5030a00b47e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.432066 30007 tablet_service.cc:1467] Processing CreateTablet for tablet e5af418a0ae043a387b5030a00b47e91 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8fad8146798f48ce9901b24b7f1944b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.432193 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5af418a0ae043a387b5030a00b47e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.432447 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5af418a0ae043a387b5030a00b47e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.435120 30006 tablet_service.cc:1467] Processing CreateTablet for tablet 1edc0a17b5f2466cbe2cf5a140689734 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8fad8146798f48ce9901b24b7f1944b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.435504 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edc0a17b5f2466cbe2cf5a140689734. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.437258 509 tablet_bootstrap.cc:492] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:09.438912 509 tablet_bootstrap.cc:654] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.440296 30138 tablet_service.cc:1467] Processing CreateTablet for tablet fefcc470084f4ec9be6ba45d3ba22ce5 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8fad8146798f48ce9901b24b7f1944b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.440408 30140 tablet_service.cc:1467] Processing CreateTablet for tablet 1edc0a17b5f2466cbe2cf5a140689734 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8fad8146798f48ce9901b24b7f1944b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.440737 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fefcc470084f4ec9be6ba45d3ba22ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.441380 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edc0a17b5f2466cbe2cf5a140689734. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.441615 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 1edc0a17b5f2466cbe2cf5a140689734 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8fad8146798f48ce9901b24b7f1944b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.442039 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edc0a17b5f2466cbe2cf5a140689734. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.446534 27499 tablet_service.cc:1467] Processing CreateTablet for tablet fefcc470084f4ec9be6ba45d3ba22ce5 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8fad8146798f48ce9901b24b7f1944b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.446921 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fefcc470084f4ec9be6ba45d3ba22ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.453356 449 tablet_bootstrap.cc:492] T 1edc0a17b5f2466cbe2cf5a140689734 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:09.455601 449 tablet_bootstrap.cc:654] T 1edc0a17b5f2466cbe2cf5a140689734 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.458415 30400 tablet_service.cc:1467] Processing CreateTablet for tablet fefcc470084f4ec9be6ba45d3ba22ce5 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8fad8146798f48ce9901b24b7f1944b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.459120 509 tablet_bootstrap.cc:492] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:09.459302 509 ts_tablet_manager.cc:1367] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.018s
I20240502 08:18:09.459594 449 tablet_bootstrap.cc:492] T 1edc0a17b5f2466cbe2cf5a140689734 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:09.459936 449 ts_tablet_manager.cc:1367] T 1edc0a17b5f2466cbe2cf5a140689734 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 08:18:09.460182 509 raft_consensus.cc:348] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:09.460413 509 raft_consensus.cc:374] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.460536 509 raft_consensus.cc:729] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.460760 449 raft_consensus.cc:348] T 1edc0a17b5f2466cbe2cf5a140689734 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:09.460826 509 consensus_queue.cc:260] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:09.461388 509 ts_tablet_manager.cc:1398] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:09.462960 456 tablet_bootstrap.cc:492] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:09.461063 449 raft_consensus.cc:374] T 1edc0a17b5f2466cbe2cf5a140689734 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.463521 449 raft_consensus.cc:729] T 1edc0a17b5f2466cbe2cf5a140689734 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.463743 449 consensus_queue.cc:260] T 1edc0a17b5f2466cbe2cf5a140689734 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:09.464359 449 ts_tablet_manager.cc:1398] T 1edc0a17b5f2466cbe2cf5a140689734 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:18:09.464529 456 tablet_bootstrap.cc:654] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.465566 449 tablet_bootstrap.cc:492] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:09.466415 27112 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:09.467010 449 tablet_bootstrap.cc:654] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.470870 512 tablet_bootstrap.cc:492] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:09.473107 512 tablet_bootstrap.cc:654] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.475445 456 tablet_bootstrap.cc:492] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:09.475647 456 ts_tablet_manager.cc:1367] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 08:18:09.477360 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fefcc470084f4ec9be6ba45d3ba22ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.477900 456 raft_consensus.cc:348] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:09.478217 456 raft_consensus.cc:374] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.478348 456 raft_consensus.cc:729] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.478307 30159 raft_consensus.cc:1261] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:09.478677 456 consensus_queue.cc:260] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:09.479023 314 consensus_queue.cc:1035] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:09.479537 456 ts_tablet_manager.cc:1398] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:18:09.479893 456 tablet_bootstrap.cc:492] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:09.479876 457 tablet_bootstrap.cc:492] T 1edc0a17b5f2466cbe2cf5a140689734 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:09.481541 456 tablet_bootstrap.cc:654] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.481559 457 tablet_bootstrap.cc:654] T 1edc0a17b5f2466cbe2cf5a140689734 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.489706 30289 raft_consensus.cc:1261] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:09.493213 506 consensus_queue.cc:1035] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:09.496613 512 tablet_bootstrap.cc:492] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:09.497824 512 ts_tablet_manager.cc:1367] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.003s
I20240502 08:18:09.499099 512 raft_consensus.cc:348] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:09.501564 512 raft_consensus.cc:374] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.502040 512 raft_consensus.cc:729] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.502976 512 consensus_queue.cc:260] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:09.508390 512 ts_tablet_manager.cc:1398] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240502 08:18:09.521736 457 tablet_bootstrap.cc:492] T 1edc0a17b5f2466cbe2cf5a140689734 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:09.521948 457 ts_tablet_manager.cc:1367] T 1edc0a17b5f2466cbe2cf5a140689734 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.006s
I20240502 08:18:09.523046 457 raft_consensus.cc:348] T 1edc0a17b5f2466cbe2cf5a140689734 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:09.523341 457 raft_consensus.cc:374] T 1edc0a17b5f2466cbe2cf5a140689734 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.523435 457 raft_consensus.cc:729] T 1edc0a17b5f2466cbe2cf5a140689734 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.523727 457 consensus_queue.cc:260] T 1edc0a17b5f2466cbe2cf5a140689734 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:09.525465 510 tablet_bootstrap.cc:492] T e5af418a0ae043a387b5030a00b47e91 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:09.525812 456 tablet_bootstrap.cc:492] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:09.526052 456 ts_tablet_manager.cc:1367] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.015s
I20240502 08:18:09.526829 509 tablet_bootstrap.cc:492] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:09.526942 456 raft_consensus.cc:348] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:09.527184 456 raft_consensus.cc:374] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.527276 456 raft_consensus.cc:729] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.527527 456 consensus_queue.cc:260] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:09.528836 509 tablet_bootstrap.cc:654] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.530788 456 ts_tablet_manager.cc:1398] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 08:18:09.534209 510 tablet_bootstrap.cc:654] T e5af418a0ae043a387b5030a00b47e91 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.543409 509 tablet_bootstrap.cc:492] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:09.543665 509 ts_tablet_manager.cc:1367] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.004s
I20240502 08:18:09.544027 457 ts_tablet_manager.cc:1398] T 1edc0a17b5f2466cbe2cf5a140689734 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.022s user 0.007s sys 0.004s
I20240502 08:18:09.544660 509 raft_consensus.cc:348] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:09.545038 509 raft_consensus.cc:374] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.545177 509 raft_consensus.cc:729] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.545516 509 consensus_queue.cc:260] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:09.549855 510 tablet_bootstrap.cc:492] T e5af418a0ae043a387b5030a00b47e91 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:09.550097 510 ts_tablet_manager.cc:1367] T e5af418a0ae043a387b5030a00b47e91 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.009s
I20240502 08:18:09.550159 390 raft_consensus.cc:482] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:09.550223 533 tablet.cc:1722] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:09.550289 390 raft_consensus.cc:504] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:09.550717 533 tablet.cc:1620] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:18:09.550781 510 raft_consensus.cc:348] T e5af418a0ae043a387b5030a00b47e91 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:09.550976 510 raft_consensus.cc:374] T e5af418a0ae043a387b5030a00b47e91 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.551043 510 raft_consensus.cc:729] T e5af418a0ae043a387b5030a00b47e91 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.551137 390 leader_election.cc:288] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:09.551206 510 consensus_queue.cc:260] T e5af418a0ae043a387b5030a00b47e91 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:09.551548 510 ts_tablet_manager.cc:1398] T e5af418a0ae043a387b5030a00b47e91 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:18:09.553344 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefcc470084f4ec9be6ba45d3ba22ce5" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:09.553812 27110 catalog_manager.cc:5576] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899) does not have the latest schema for tablet 125f21b7543143f5ba6c52a1385c37d5 (table rebalancer_dynamic_table [id=3197a001f13546fbb549cdb7db5750c2]). Expected version 1 got 0
I20240502 08:18:09.563215 27112 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=3197a001f13546fbb549cdb7db5750c2] alter complete (version 1)
I20240502 08:18:09.565433 449 tablet_bootstrap.cc:492] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:09.565574 449 ts_tablet_manager.cc:1367] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.100s user 0.008s sys 0.007s
I20240502 08:18:09.566332 449 raft_consensus.cc:348] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:09.566622 449 raft_consensus.cc:374] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.566763 449 raft_consensus.cc:729] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.566980 27516 raft_consensus.cc:1203] T 0a5a0175799e478c9f53ecc82ba090ae P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:09.567039 449 consensus_queue.cc:260] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:09.567677 449 ts_tablet_manager.cc:1398] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:09.571630 532 tablet.cc:1722] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:09.572793 27110 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet 125f21b7543143f5ba6c52a1385c37d5 (table rebalancer_dynamic_table [id=3197a001f13546fbb549cdb7db5750c2]). Expected version 1 got 0
I20240502 08:18:09.576388 532 tablet.cc:1620] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:09.579058 390 raft_consensus.cc:482] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:09.579241 390 raft_consensus.cc:504] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:09.579921 390 leader_election.cc:288] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:09.580229 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5af418a0ae043a387b5030a00b47e91" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:09.580456 27258 raft_consensus.cc:2454] T e5af418a0ae043a387b5030a00b47e91 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:09.580492 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5af418a0ae043a387b5030a00b47e91" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:09.580680 30027 raft_consensus.cc:2454] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:09.581104 30357 leader_election.cc:302] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters:
I20240502 08:18:09.581568 390 raft_consensus.cc:2781] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:09.581650 390 raft_consensus.cc:482] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:09.581741 390 raft_consensus.cc:3037] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.584203 390 raft_consensus.cc:504] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:09.585377 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5af418a0ae043a387b5030a00b47e91" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:09.585549 30027 raft_consensus.cc:3037] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.585505 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5af418a0ae043a387b5030a00b47e91" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:09.585675 27258 raft_consensus.cc:3037] T e5af418a0ae043a387b5030a00b47e91 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.585793 535 tablet.cc:1722] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:09.586385 535 tablet.cc:1620] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:09.588224 30027 raft_consensus.cc:2454] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:09.588335 27258 raft_consensus.cc:2454] T e5af418a0ae043a387b5030a00b47e91 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:09.588476 390 leader_election.cc:288] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:09.588933 30355 leader_election.cc:302] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters:
I20240502 08:18:09.589293 390 raft_consensus.cc:2781] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:09.589860 509 ts_tablet_manager.cc:1398] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.046s user 0.000s sys 0.001s
W20240502 08:18:09.599598 30358 leader_election.cc:341] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:09.602284 27111 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet 125f21b7543143f5ba6c52a1385c37d5 (table rebalancer_dynamic_table [id=3197a001f13546fbb549cdb7db5750c2]). Expected version 1 got 0
I20240502 08:18:09.603855 506 raft_consensus.cc:482] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:09.604007 506 raft_consensus.cc:504] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:09.604629 506 leader_election.cc:288] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:09.606992 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edc0a17b5f2466cbe2cf5a140689734" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:09.607219 30289 raft_consensus.cc:2454] T 1edc0a17b5f2466cbe2cf5a140689734 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:09.607571 29961 leader_election.cc:302] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:18:09.607710 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edc0a17b5f2466cbe2cf5a140689734" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:09.607865 506 raft_consensus.cc:2781] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:09.607946 506 raft_consensus.cc:482] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:09.607978 30159 raft_consensus.cc:2454] T 1edc0a17b5f2466cbe2cf5a140689734 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:09.608040 506 raft_consensus.cc:3037] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.610777 390 raft_consensus.cc:686] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:18:09.610858 506 raft_consensus.cc:504] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:09.611193 390 consensus_queue.cc:237] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:09.611950 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edc0a17b5f2466cbe2cf5a140689734" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:09.612200 30159 raft_consensus.cc:3037] T 1edc0a17b5f2466cbe2cf5a140689734 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.612156 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edc0a17b5f2466cbe2cf5a140689734" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:09.612269 506 leader_election.cc:288] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:09.612390 30288 raft_consensus.cc:3037] T 1edc0a17b5f2466cbe2cf5a140689734 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.612778 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefcc470084f4ec9be6ba45d3ba22ce5" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:18:09.612972 27516 raft_consensus.cc:2454] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:09.613405 30358 leader_election.cc:302] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, b01e7c8a302d41078fa39cfef665ff59; no voters: 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:09.615090 30159 raft_consensus.cc:2454] T 1edc0a17b5f2466cbe2cf5a140689734 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:09.615556 29964 leader_election.cc:302] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:09.615871 506 raft_consensus.cc:2781] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:09.616344 506 raft_consensus.cc:686] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:18:09.616719 30288 raft_consensus.cc:2454] T 1edc0a17b5f2466cbe2cf5a140689734 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:09.616638 506 consensus_queue.cc:237] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:09.617738 537 raft_consensus.cc:2781] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:09.617882 537 raft_consensus.cc:482] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:09.618086 537 raft_consensus.cc:3037] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.621596 537 raft_consensus.cc:504] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:09.622494 537 leader_election.cc:288] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:09.622382 27112 catalog_manager.cc:5484] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:09.622738 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefcc470084f4ec9be6ba45d3ba22ce5" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:09.622906 30159 raft_consensus.cc:3037] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.625417 30159 raft_consensus.cc:2454] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:09.626351 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefcc470084f4ec9be6ba45d3ba22ce5" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:18:09.626529 27516 raft_consensus.cc:3037] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.629773 27516 raft_consensus.cc:2454] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:09.632071 30358 leader_election.cc:302] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:09.632469 390 raft_consensus.cc:2781] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:09.632611 390 raft_consensus.cc:686] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:18:09.633005 390 consensus_queue.cc:237] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:09.637204 27112 catalog_manager.cc:5484] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:09.637246 27111 catalog_manager.cc:5484] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:09.680061 27112 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:09.686686 532 tablet.cc:1722] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:09.687157 532 tablet.cc:1620] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:09.687276 533 tablet.cc:1722] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:09.687696 533 tablet.cc:1620] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:18:09.688323 535 tablet.cc:1722] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:09.688766 535 tablet.cc:1620] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:09.697795 27112 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=3197a001f13546fbb549cdb7db5750c2] alter complete (version 2)
I20240502 08:18:09.757207 27258 consensus_queue.cc:237] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:09.759012 30288 raft_consensus.cc:1261] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:18:09.759691 32175 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:09.760316 27648 raft_consensus.cc:1261] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:18:09.760774 32175 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:09.762001 32462 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:09.762558 30288 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:09.763775 27648 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:09.766868 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 903502e57ab2430383807b0d46a39847 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:09.769702 27111 catalog_manager.cc:5484] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 5 to 7, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 2 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:09.773524 27499 tablet_service.cc:1514] Processing DeleteTablet for tablet 903502e57ab2430383807b0d46a39847 with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:09.786981 557 tablet_replica.cc:330] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:09.787328 557 raft_consensus.cc:2229] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:09.787546 557 raft_consensus.cc:2258] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:09.788529 557 ts_tablet_manager.cc:1875] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:09.793366 30028 raft_consensus.cc:1203] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:09.799186 557 ts_tablet_manager.cc:1888] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:18:09.799271 557 log.cc:1196] T 903502e57ab2430383807b0d46a39847 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/903502e57ab2430383807b0d46a39847
I20240502 08:18:09.799886 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 903502e57ab2430383807b0d46a39847 (table rebalance_test_table_13 [id=28a67fad5bff471981e8cdf2b2a2b34a]) successfully deleted
I20240502 08:18:09.800740 27112 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:09.807780 532 tablet.cc:1722] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:09.808526 535 tablet.cc:1722] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:09.810070 533 tablet.cc:1722] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:09.818228 27112 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=3197a001f13546fbb549cdb7db5750c2] alter complete (version 3)
I20240502 08:18:09.922675 27112 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:09.922878 27112 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:09.926077 27112 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 125f21b7543143f5ba6c52a1385c37d5
I20240502 08:18:09.926862 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet 125f21b7543143f5ba6c52a1385c37d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:09 UTC) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:09.927063 30007 tablet_service.cc:1514] Processing DeleteTablet for tablet 125f21b7543143f5ba6c52a1385c37d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:09 UTC) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:09.927443 30140 tablet_service.cc:1514] Processing DeleteTablet for tablet 125f21b7543143f5ba6c52a1385c37d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:09 UTC) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:09.927886 583 tablet_replica.cc:330] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:09.927983 585 tablet_replica.cc:330] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:09.928197 585 raft_consensus.cc:2229] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:09.928349 583 raft_consensus.cc:2229] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:09.928400 585 raft_consensus.cc:2258] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:09.928763 583 raft_consensus.cc:2258] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:09.929371 585 ts_tablet_manager.cc:1875] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:09.930068 583 ts_tablet_manager.cc:1875] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:09.933703 585 ts_tablet_manager.cc:1888] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:09.933810 585 log.cc:1196] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/125f21b7543143f5ba6c52a1385c37d5
I20240502 08:18:09.934258 585 ts_tablet_manager.cc:1909] T 125f21b7543143f5ba6c52a1385c37d5 P 84cda20738d2452da166e31fc6da57a6: Deleting consensus metadata
I20240502 08:18:09.935127 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet 125f21b7543143f5ba6c52a1385c37d5 (table rebalancer_dynamic_table_new_name [id=3197a001f13546fbb549cdb7db5750c2]) successfully deleted
I20240502 08:18:09.938313 584 tablet_replica.cc:330] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:09.938503 584 raft_consensus.cc:2229] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:09.938761 583 ts_tablet_manager.cc:1888] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:09.938799 584 raft_consensus.cc:2258] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:09.938879 583 log.cc:1196] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/125f21b7543143f5ba6c52a1385c37d5
I20240502 08:18:09.939270 583 ts_tablet_manager.cc:1909] T 125f21b7543143f5ba6c52a1385c37d5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting consensus metadata
I20240502 08:18:09.939481 584 ts_tablet_manager.cc:1875] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:09.940204 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet 125f21b7543143f5ba6c52a1385c37d5 (table rebalancer_dynamic_table_new_name [id=3197a001f13546fbb549cdb7db5750c2]) successfully deleted
I20240502 08:18:09.944589 584 ts_tablet_manager.cc:1888] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:09.944662 32437 raft_consensus.cc:1050] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6: attempting to promote NON_VOTER b01e7c8a302d41078fa39cfef665ff59 to VOTER
I20240502 08:18:09.944767 584 log.cc:1196] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/125f21b7543143f5ba6c52a1385c37d5
I20240502 08:18:09.945266 584 ts_tablet_manager.cc:1909] T 125f21b7543143f5ba6c52a1385c37d5 P 1f0bf4bc3c5146539146b825b048bff3: Deleting consensus metadata
I20240502 08:18:09.945590 32437 consensus_queue.cc:237] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:09.946303 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet 125f21b7543143f5ba6c52a1385c37d5 (table rebalancer_dynamic_table_new_name [id=3197a001f13546fbb549cdb7db5750c2]) successfully deleted
I20240502 08:18:09.948007 30421 raft_consensus.cc:1261] T 0a5a0175799e478c9f53ecc82ba090ae P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:09.948104 30288 raft_consensus.cc:1261] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:09.948642 27519 raft_consensus.cc:1261] T 0a5a0175799e478c9f53ecc82ba090ae P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:09.948596 32437 consensus_queue.cc:1035] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:09.949106 587 consensus_queue.cc:1035] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:09.950069 32437 consensus_queue.cc:1035] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:09.955200 587 raft_consensus.cc:2932] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:09.958168 30422 raft_consensus.cc:2932] T 0a5a0175799e478c9f53ecc82ba090ae P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:09.961741 27112 catalog_manager.cc:5484] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:09.962110 30288 raft_consensus.cc:2932] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:09.966360 30159 consensus_queue.cc:237] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:09.968053 30421 raft_consensus.cc:1261] T 0a5a0175799e478c9f53ecc82ba090ae P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:09.967947 27519 raft_consensus.cc:2932] T 0a5a0175799e478c9f53ecc82ba090ae P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:09.968525 587 consensus_queue.cc:1035] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:09.970499 32437 raft_consensus.cc:2932] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:09.971390 30421 raft_consensus.cc:2932] T 0a5a0175799e478c9f53ecc82ba090ae P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:09.973004 27516 raft_consensus.cc:1261] T 0a5a0175799e478c9f53ecc82ba090ae P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:09.973918 587 consensus_queue.cc:1035] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:09.975134 27516 raft_consensus.cc:2932] T 0a5a0175799e478c9f53ecc82ba090ae P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:09.977061 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0a5a0175799e478c9f53ecc82ba090ae with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:09.980392 27112 catalog_manager.cc:5484] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:09.984516 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5a0175799e478c9f53ecc82ba090ae with delete_type TABLET_DATA_TOMBSTONED (TS 49c5c8f70e494edd9fcda40cc0af6cc4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:09.986042 583 tablet_replica.cc:330] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:09.986284 583 raft_consensus.cc:2229] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:09.986521 583 raft_consensus.cc:2258] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:09.987272 583 ts_tablet_manager.cc:1875] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:09.991660 583 ts_tablet_manager.cc:1888] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:09.991812 583 log.cc:1196] T 0a5a0175799e478c9f53ecc82ba090ae P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/0a5a0175799e478c9f53ecc82ba090ae
I20240502 08:18:09.992466 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet 0a5a0175799e478c9f53ecc82ba090ae (table rebalancer_extra_table_10 [id=9df4632350e6444089d25e0f410be77a]) successfully deleted
I20240502 08:18:10.028739 27111 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:10.035424 30400 tablet_service.cc:1467] Processing CreateTablet for tablet 24adf1f8ca494d56abaeab3d64d0cbb0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c24f28fd56994954963b460253ff150a]), partition=
I20240502 08:18:10.035555 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 24adf1f8ca494d56abaeab3d64d0cbb0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c24f28fd56994954963b460253ff150a]), partition=
I20240502 08:18:10.035802 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24adf1f8ca494d56abaeab3d64d0cbb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.035923 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24adf1f8ca494d56abaeab3d64d0cbb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.035941 30140 tablet_service.cc:1467] Processing CreateTablet for tablet 24adf1f8ca494d56abaeab3d64d0cbb0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c24f28fd56994954963b460253ff150a]), partition=
I20240502 08:18:10.036302 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24adf1f8ca494d56abaeab3d64d0cbb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.041281 509 tablet_bootstrap.cc:492] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:10.042624 509 tablet_bootstrap.cc:654] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.042904 449 tablet_bootstrap.cc:492] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:10.043535 457 tablet_bootstrap.cc:492] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:10.044342 449 tablet_bootstrap.cc:654] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.044737 457 tablet_bootstrap.cc:654] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.046048 449 tablet_bootstrap.cc:492] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:10.046202 449 ts_tablet_manager.cc:1367] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:10.046804 509 tablet_bootstrap.cc:492] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:10.046932 509 ts_tablet_manager.cc:1367] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:18:10.046891 449 raft_consensus.cc:348] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:10.047108 449 raft_consensus.cc:374] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.047169 449 raft_consensus.cc:729] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.047232 457 tablet_bootstrap.cc:492] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:10.047343 457 ts_tablet_manager.cc:1367] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:18:10.047406 449 consensus_queue.cc:260] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:10.047650 509 raft_consensus.cc:348] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:10.047819 509 raft_consensus.cc:374] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.047899 509 raft_consensus.cc:729] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.047983 449 ts_tablet_manager.cc:1398] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:10.047910 457 raft_consensus.cc:348] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:10.048089 457 raft_consensus.cc:374] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.048211 457 raft_consensus.cc:729] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.048142 509 consensus_queue.cc:260] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:10.048426 457 consensus_queue.cc:260] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:10.048760 509 ts_tablet_manager.cc:1398] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:10.049008 457 ts_tablet_manager.cc:1398] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:10.133627 554 consensus_queue.cc:1035] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:10.139070 506 consensus_queue.cc:1035] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:10.165355 27112 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:10.173004 390 consensus_queue.cc:1035] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:10.173687 30400 tablet_service.cc:1467] Processing CreateTablet for tablet 1adfeb89dc5341a2b8cf962eb28da45e (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=3c7f8803df8a4ae885409a0c0a114f1f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.174180 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1adfeb89dc5341a2b8cf962eb28da45e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.174268 30401 tablet_service.cc:1467] Processing CreateTablet for tablet 70f4c40bc22947468dfd15804a4e66cd (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=3c7f8803df8a4ae885409a0c0a114f1f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.174660 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f4c40bc22947468dfd15804a4e66cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.178647 30007 tablet_service.cc:1467] Processing CreateTablet for tablet 70f4c40bc22947468dfd15804a4e66cd (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=3c7f8803df8a4ae885409a0c0a114f1f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.179075 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f4c40bc22947468dfd15804a4e66cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.179425 30140 tablet_service.cc:1467] Processing CreateTablet for tablet 1adfeb89dc5341a2b8cf962eb28da45e (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=3c7f8803df8a4ae885409a0c0a114f1f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.179828 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1adfeb89dc5341a2b8cf962eb28da45e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.180050 30138 tablet_service.cc:1467] Processing CreateTablet for tablet 757f65cd77bf49d882c2545673282e03 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=3c7f8803df8a4ae885409a0c0a114f1f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.180411 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757f65cd77bf49d882c2545673282e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.182880 27498 tablet_service.cc:1467] Processing CreateTablet for tablet 757f65cd77bf49d882c2545673282e03 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=3c7f8803df8a4ae885409a0c0a114f1f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.183291 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757f65cd77bf49d882c2545673282e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.185303 449 tablet_bootstrap.cc:492] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:10.185531 27499 tablet_service.cc:1467] Processing CreateTablet for tablet 1adfeb89dc5341a2b8cf962eb28da45e (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=3c7f8803df8a4ae885409a0c0a114f1f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.185850 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1adfeb89dc5341a2b8cf962eb28da45e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.186849 449 tablet_bootstrap.cc:654] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.189239 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 757f65cd77bf49d882c2545673282e03 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=3c7f8803df8a4ae885409a0c0a114f1f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.189807 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757f65cd77bf49d882c2545673282e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.190451 27500 tablet_service.cc:1467] Processing CreateTablet for tablet 70f4c40bc22947468dfd15804a4e66cd (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=3c7f8803df8a4ae885409a0c0a114f1f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.190847 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f4c40bc22947468dfd15804a4e66cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.194088 509 tablet_bootstrap.cc:492] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:10.195868 509 tablet_bootstrap.cc:654] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.199764 613 tablet_bootstrap.cc:492] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:10.209239 613 tablet_bootstrap.cc:654] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.217854 613 tablet_bootstrap.cc:492] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:10.218593 613 ts_tablet_manager.cc:1367] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20240502 08:18:10.219277 509 tablet_bootstrap.cc:492] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:10.219502 509 ts_tablet_manager.cc:1367] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240502 08:18:10.220360 613 raft_consensus.cc:348] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.220593 613 raft_consensus.cc:374] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.220417 509 raft_consensus.cc:348] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.220932 509 raft_consensus.cc:374] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.221060 509 raft_consensus.cc:729] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.221400 509 consensus_queue.cc:260] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.220696 613 raft_consensus.cc:729] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.224372 613 consensus_queue.cc:260] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.227140 613 ts_tablet_manager.cc:1398] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 08:18:10.227588 525 raft_consensus.cc:482] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:10.227747 525 raft_consensus.cc:504] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:10.228622 525 leader_election.cc:288] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:10.229445 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24adf1f8ca494d56abaeab3d64d0cbb0" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:10.229650 30159 raft_consensus.cc:2454] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:10.230078 30226 leader_election.cc:302] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:10.230466 525 raft_consensus.cc:2781] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:10.230571 525 raft_consensus.cc:482] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:10.230667 525 raft_consensus.cc:3037] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.233186 525 raft_consensus.cc:504] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:10.235507 390 consensus_queue.cc:1035] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:10.236063 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24adf1f8ca494d56abaeab3d64d0cbb0" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:10.236256 30159 raft_consensus.cc:3037] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.240553 30159 raft_consensus.cc:2454] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:10.240777 525 leader_election.cc:288] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:10.241312 30226 leader_election.cc:302] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:10.241370 509 ts_tablet_manager.cc:1398] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240502 08:18:10.241892 525 raft_consensus.cc:2781] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:10.245275 509 tablet_bootstrap.cc:492] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:10.245370 457 tablet_bootstrap.cc:492] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:10.247093 457 tablet_bootstrap.cc:654] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.254395 390 raft_consensus.cc:482] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:10.254581 390 raft_consensus.cc:504] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:10.254822 525 raft_consensus.cc:686] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:18:10.255121 525 consensus_queue.cc:237] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:10.255509 390 leader_election.cc:288] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:10.256065 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24adf1f8ca494d56abaeab3d64d0cbb0" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:10.256362 30159 raft_consensus.cc:2379] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in current term 1: Already voted for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in this term.
I20240502 08:18:10.256793 509 tablet_bootstrap.cc:654] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.275573 449 tablet_bootstrap.cc:492] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:10.275732 449 ts_tablet_manager.cc:1367] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.091s user 0.008s sys 0.010s
I20240502 08:18:10.276683 449 raft_consensus.cc:348] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.276971 449 raft_consensus.cc:374] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.277067 449 raft_consensus.cc:729] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.277670 449 consensus_queue.cc:260] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.283028 449 ts_tablet_manager.cc:1398] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240502 08:18:10.283756 509 tablet_bootstrap.cc:492] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:10.283864 621 tablet_bootstrap.cc:492] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:10.283998 509 ts_tablet_manager.cc:1367] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20240502 08:18:10.285804 621 tablet_bootstrap.cc:654] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.286384 509 raft_consensus.cc:348] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.286650 509 raft_consensus.cc:374] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.286780 509 raft_consensus.cc:729] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.287173 628 consensus_queue.cc:1035] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:10.288491 509 consensus_queue.cc:260] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.289132 509 ts_tablet_manager.cc:1398] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:10.289722 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24adf1f8ca494d56abaeab3d64d0cbb0" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:10.290781 27112 catalog_manager.cc:5484] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7). New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:10.294405 30355 leader_election.cc:302] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:10.297116 630 raft_consensus.cc:3037] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.297974 457 tablet_bootstrap.cc:492] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:10.299322 457 ts_tablet_manager.cc:1367] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.005s
I20240502 08:18:10.299582 449 tablet_bootstrap.cc:492] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:10.301971 630 raft_consensus.cc:2726] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:10.303092 457 raft_consensus.cc:348] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.303385 457 raft_consensus.cc:374] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.303484 457 raft_consensus.cc:729] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.303766 457 consensus_queue.cc:260] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.304836 457 ts_tablet_manager.cc:1398] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:18:10.305619 449 tablet_bootstrap.cc:654] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.317389 449 tablet_bootstrap.cc:492] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:10.317653 449 ts_tablet_manager.cc:1367] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20240502 08:18:10.318748 449 raft_consensus.cc:348] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.319559 449 raft_consensus.cc:374] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.319690 449 raft_consensus.cc:729] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.321512 449 consensus_queue.cc:260] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.322204 449 ts_tablet_manager.cc:1398] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 08:18:10.322772 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24adf1f8ca494d56abaeab3d64d0cbb0" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:10.323004 30421 raft_consensus.cc:2454] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:10.323557 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24adf1f8ca494d56abaeab3d64d0cbb0" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:10.326318 30421 raft_consensus.cc:2454] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:10.327981 616 consensus_queue.cc:1035] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:10.329103 621 tablet_bootstrap.cc:492] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:10.329233 621 ts_tablet_manager.cc:1367] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.003s
I20240502 08:18:10.330183 621 raft_consensus.cc:348] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.330412 621 raft_consensus.cc:374] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.330469 621 raft_consensus.cc:729] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.330703 621 consensus_queue.cc:260] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.331609 621 ts_tablet_manager.cc:1398] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:10.331912 621 tablet_bootstrap.cc:492] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:10.334085 621 tablet_bootstrap.cc:654] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.342067 621 tablet_bootstrap.cc:492] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:10.342233 621 ts_tablet_manager.cc:1367] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 08:18:10.342861 621 raft_consensus.cc:348] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.343050 616 raft_consensus.cc:482] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:10.343192 621 raft_consensus.cc:374] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.343257 621 raft_consensus.cc:729] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.343250 616 raft_consensus.cc:504] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.343446 621 consensus_queue.cc:260] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.344035 621 ts_tablet_manager.cc:1398] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:10.344118 616 leader_election.cc:288] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:10.344261 621 tablet_bootstrap.cc:492] T 1adfeb89dc5341a2b8cf962eb28da45e P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:10.345183 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1adfeb89dc5341a2b8cf962eb28da45e" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:18:10.346031 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1adfeb89dc5341a2b8cf962eb28da45e" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:10.346315 30159 raft_consensus.cc:2454] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:10.345535 621 tablet_bootstrap.cc:654] T 1adfeb89dc5341a2b8cf962eb28da45e P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.347119 30358 leader_election.cc:302] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
W20240502 08:18:10.347466 30358 leader_election.cc:341] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:10.347862 616 raft_consensus.cc:2781] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:10.347976 616 raft_consensus.cc:482] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:10.348064 616 raft_consensus.cc:3037] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.348671 621 tablet_bootstrap.cc:492] T 1adfeb89dc5341a2b8cf962eb28da45e P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:10.348752 621 ts_tablet_manager.cc:1367] T 1adfeb89dc5341a2b8cf962eb28da45e P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:10.349144 621 raft_consensus.cc:348] T 1adfeb89dc5341a2b8cf962eb28da45e P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.349283 621 raft_consensus.cc:374] T 1adfeb89dc5341a2b8cf962eb28da45e P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.349329 621 raft_consensus.cc:729] T 1adfeb89dc5341a2b8cf962eb28da45e P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.349483 621 consensus_queue.cc:260] T 1adfeb89dc5341a2b8cf962eb28da45e P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.349830 621 ts_tablet_manager.cc:1398] T 1adfeb89dc5341a2b8cf962eb28da45e P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:18:10.355742 616 raft_consensus.cc:504] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.357856 616 leader_election.cc:288] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:10.360199 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1adfeb89dc5341a2b8cf962eb28da45e" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:18:10.360435 27516 raft_consensus.cc:3037] T 1adfeb89dc5341a2b8cf962eb28da45e P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.360909 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1adfeb89dc5341a2b8cf962eb28da45e" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:10.361125 30159 raft_consensus.cc:3037] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.363742 27516 raft_consensus.cc:2454] T 1adfeb89dc5341a2b8cf962eb28da45e P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:10.366950 30358 leader_election.cc:302] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:10.367457 616 raft_consensus.cc:2781] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:10.367602 616 raft_consensus.cc:686] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:18:10.367834 616 consensus_queue.cc:237] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.373582 27112 catalog_manager.cc:5484] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:10.375309 32721 raft_consensus.cc:1050] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 to VOTER
I20240502 08:18:10.376160 32721 consensus_queue.cc:237] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:10.378144 30159 raft_consensus.cc:2454] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:10.407435 30027 raft_consensus.cc:1261] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:10.407970 485 consensus_queue.cc:1035] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:10.416777 27257 raft_consensus.cc:1261] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:10.417516 485 consensus_queue.cc:1035] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:10.432896 27648 raft_consensus.cc:1261] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:10.433698 485 consensus_queue.cc:1035] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:10.437721 27111 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:10.439610 32721 raft_consensus.cc:2932] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:10.442152 27257 raft_consensus.cc:2932] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:10.444801 30159 raft_consensus.cc:1261] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:10.445710 30421 raft_consensus.cc:1261] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:10.446451 624 consensus_queue.cc:1035] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:10.448004 624 consensus_queue.cc:1035] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:10.455202 27111 catalog_manager.cc:5484] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:10.464781 27648 raft_consensus.cc:2932] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:10.485194 506 raft_consensus.cc:482] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:10.485340 506 raft_consensus.cc:504] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.486058 506 leader_election.cc:288] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:10.486769 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f4c40bc22947468dfd15804a4e66cd" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:18:10.486963 27516 raft_consensus.cc:2454] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:10.487414 29964 leader_election.cc:302] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:10.487756 506 raft_consensus.cc:2781] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:10.487838 506 raft_consensus.cc:482] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:10.487921 506 raft_consensus.cc:3037] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.492337 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f4c40bc22947468dfd15804a4e66cd" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:10.492550 30422 raft_consensus.cc:2454] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:10.493322 506 raft_consensus.cc:504] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.495988 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f4c40bc22947468dfd15804a4e66cd" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:18:10.496146 27516 raft_consensus.cc:3037] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.499801 506 leader_election.cc:288] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:10.500625 27516 raft_consensus.cc:2454] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:10.501052 29964 leader_election.cc:302] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:10.501509 506 raft_consensus.cc:2781] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:10.501816 506 raft_consensus.cc:686] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:18:10.501818 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f4c40bc22947468dfd15804a4e66cd" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:10.502056 30422 raft_consensus.cc:3037] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.502156 506 consensus_queue.cc:237] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.504714 30422 raft_consensus.cc:2454] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:10.506892 30027 raft_consensus.cc:2932] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:10.515117 27111 catalog_manager.cc:5484] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:10.518657 27110 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet 24adf1f8ca494d56abaeab3d64d0cbb0 (table rebalancer_dynamic_table [id=c24f28fd56994954963b460253ff150a]). Expected version 1 got 0
I20240502 08:18:10.519876 27112 catalog_manager.cc:5576] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899) does not have the latest schema for tablet 24adf1f8ca494d56abaeab3d64d0cbb0 (table rebalancer_dynamic_table [id=c24f28fd56994954963b460253ff150a]). Expected version 1 got 0
I20240502 08:18:10.524905 657 tablet.cc:1722] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:10.525485 657 tablet.cc:1620] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:10.526502 27112 catalog_manager.cc:5576] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899) does not have the latest schema for tablet 24adf1f8ca494d56abaeab3d64d0cbb0 (table rebalancer_dynamic_table [id=c24f28fd56994954963b460253ff150a]). Expected version 1 got 0
I20240502 08:18:10.527485 27111 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet 24adf1f8ca494d56abaeab3d64d0cbb0 (table rebalancer_dynamic_table [id=c24f28fd56994954963b460253ff150a]). Expected version 1 got 0
I20240502 08:18:10.530627 659 tablet.cc:1722] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:10.532295 658 tablet.cc:1722] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:10.532837 658 tablet.cc:1620] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:18:10.532845 27112 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet 24adf1f8ca494d56abaeab3d64d0cbb0 (table rebalancer_dynamic_table [id=c24f28fd56994954963b460253ff150a]). Expected version 1 got 0
I20240502 08:18:10.533418 659 tablet.cc:1620] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:10.540452 27111 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=c24f28fd56994954963b460253ff150a] alter complete (version 1)
I20240502 08:18:10.544713 626 raft_consensus.cc:482] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:10.544999 626 raft_consensus.cc:504] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.546463 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757f65cd77bf49d882c2545673282e03" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:10.546757 30157 raft_consensus.cc:2454] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:10.547227 30226 leader_election.cc:302] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:10.547334 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757f65cd77bf49d882c2545673282e03" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:18:10.547509 27516 raft_consensus.cc:2454] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:10.547773 623 raft_consensus.cc:2781] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:10.547852 623 raft_consensus.cc:482] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:10.547938 623 raft_consensus.cc:3037] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.551638 623 raft_consensus.cc:504] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.553524 623 leader_election.cc:288] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:10.553725 626 leader_election.cc:288] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:10.554169 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757f65cd77bf49d882c2545673282e03" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:18:10.554329 27516 raft_consensus.cc:3037] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.554263 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757f65cd77bf49d882c2545673282e03" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:10.554476 30157 raft_consensus.cc:3037] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.556399 27516 raft_consensus.cc:2454] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:10.556830 30226 leader_election.cc:302] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:10.556952 30157 raft_consensus.cc:2454] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:10.557148 626 raft_consensus.cc:2781] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:10.557360 626 raft_consensus.cc:686] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:18:10.557806 626 consensus_queue.cc:237] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:10.560572 27516 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9725e08a6c5d4410b8d944ab154ec1eb"
dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60828
I20240502 08:18:10.560796 27516 raft_consensus.cc:595] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:18:10.561626 27112 catalog_manager.cc:5484] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7). New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:10.655756 27112 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:10.668551 657 tablet.cc:1722] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:10.668843 658 tablet.cc:1722] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:10.669056 657 tablet.cc:1620] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:10.669382 658 tablet.cc:1620] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6: MemRowSet was empty: no flush needed.
I20240502 08:18:10.677598 659 tablet.cc:1722] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:10.678099 659 tablet.cc:1620] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:10.683216 27112 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=c24f28fd56994954963b460253ff150a] alter complete (version 2)
I20240502 08:18:10.788040 27112 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:10.794100 27516 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9725e08a6c5d4410b8d944ab154ec1eb"
dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60842
I20240502 08:18:10.794307 27516 raft_consensus.cc:595] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:18:10.796520 657 tablet.cc:1722] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:10.798148 658 tablet.cc:1722] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:10.798728 659 tablet.cc:1722] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:10.804869 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=c24f28fd56994954963b460253ff150a] alter complete (version 3)
I20240502 08:18:10.867048 650 raft_consensus.cc:979] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59: : Instructing follower bf8ac71b15f740a0a928911360773442 to start an election
I20240502 08:18:10.867329 650 raft_consensus.cc:1067] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Signalling peer bf8ac71b15f740a0a928911360773442 to start an election
I20240502 08:18:10.867969 27648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9725e08a6c5d4410b8d944ab154ec1eb"
dest_uuid: "bf8ac71b15f740a0a928911360773442"
from {username='slave'} at 127.20.207.3:50699
I20240502 08:18:10.868196 27648 raft_consensus.cc:482] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:10.868304 27648 raft_consensus.cc:3037] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:10.870747 27648 raft_consensus.cc:504] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:10.871488 27648 leader_election.cc:288] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:10.871896 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9725e08a6c5d4410b8d944ab154ec1eb" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:10.871927 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9725e08a6c5d4410b8d944ab154ec1eb" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:10.872079 30027 raft_consensus.cc:3037] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:10.872144 27257 raft_consensus.cc:3037] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:10.873834 30027 raft_consensus.cc:2454] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 2.
I20240502 08:18:10.873911 27257 raft_consensus.cc:2454] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 2.
I20240502 08:18:10.874344 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9725e08a6c5d4410b8d944ab154ec1eb" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:18:10.874436 27586 leader_election.cc:302] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 1f0bf4bc3c5146539146b825b048bff3, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:10.874545 27516 raft_consensus.cc:3032] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:10.874631 27516 raft_consensus.cc:729] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:18:10.874743 639 raft_consensus.cc:2781] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:10.874883 639 raft_consensus.cc:686] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [term 2 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:18:10.874884 27516 consensus_queue.cc:260] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:10.875308 27516 raft_consensus.cc:3037] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:10.875152 639 consensus_queue.cc:237] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:10.877815 30027 raft_consensus.cc:1226] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Rejecting Update request from peer b01e7c8a302d41078fa39cfef665ff59 for earlier term 1. Current term is 2. Ops: []
I20240502 08:18:10.878015 27516 raft_consensus.cc:2454] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 2.
I20240502 08:18:10.878713 27110 catalog_manager.cc:5484] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 1 to 2, leader changed from b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 2 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:10.906857 27110 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:10.907047 27110 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:10.910094 27110 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 24adf1f8ca494d56abaeab3d64d0cbb0
I20240502 08:18:10.910843 30401 tablet_service.cc:1514] Processing DeleteTablet for tablet 24adf1f8ca494d56abaeab3d64d0cbb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:10 UTC) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:10.911191 30140 tablet_service.cc:1514] Processing DeleteTablet for tablet 24adf1f8ca494d56abaeab3d64d0cbb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:10 UTC) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:10.911280 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet 24adf1f8ca494d56abaeab3d64d0cbb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:10 UTC) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:10.911608 676 tablet_replica.cc:330] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:10.911845 676 raft_consensus.cc:2229] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:10.911872 677 tablet_replica.cc:330] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:10.912060 676 raft_consensus.cc:2258] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:10.912155 677 raft_consensus.cc:2229] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:10.912441 677 raft_consensus.cc:2258] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:10.912875 676 ts_tablet_manager.cc:1875] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:10.913525 677 ts_tablet_manager.cc:1875] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:10.916592 390 consensus_queue.cc:1035] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:10.918641 676 ts_tablet_manager.cc:1888] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:10.919582 676 log.cc:1196] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/24adf1f8ca494d56abaeab3d64d0cbb0
I20240502 08:18:10.920295 676 ts_tablet_manager.cc:1909] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting consensus metadata
I20240502 08:18:10.922092 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet 24adf1f8ca494d56abaeab3d64d0cbb0 (table rebalancer_dynamic_table_new_name [id=c24f28fd56994954963b460253ff150a]) successfully deleted
I20240502 08:18:10.922288 677 ts_tablet_manager.cc:1888] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:10.922435 677 log.cc:1196] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/24adf1f8ca494d56abaeab3d64d0cbb0
I20240502 08:18:10.922618 678 tablet_replica.cc:330] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:10.922960 677 ts_tablet_manager.cc:1909] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 84cda20738d2452da166e31fc6da57a6: Deleting consensus metadata
I20240502 08:18:10.922957 678 raft_consensus.cc:2229] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:10.923455 678 raft_consensus.cc:2258] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:10.924187 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet 24adf1f8ca494d56abaeab3d64d0cbb0 (table rebalancer_dynamic_table_new_name [id=c24f28fd56994954963b460253ff150a]) successfully deleted
I20240502 08:18:10.924325 678 ts_tablet_manager.cc:1875] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:10.929366 678 ts_tablet_manager.cc:1888] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:10.929486 678 log.cc:1196] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/24adf1f8ca494d56abaeab3d64d0cbb0
I20240502 08:18:10.929965 678 ts_tablet_manager.cc:1909] T 24adf1f8ca494d56abaeab3d64d0cbb0 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting consensus metadata
I20240502 08:18:10.932332 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet 24adf1f8ca494d56abaeab3d64d0cbb0 (table rebalancer_dynamic_table_new_name [id=c24f28fd56994954963b460253ff150a]) successfully deleted
I20240502 08:18:10.932461 632 consensus_queue.cc:1035] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.012344 27112 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:11.018774 27631 tablet_service.cc:1467] Processing CreateTablet for tablet ad09fc1b096c4bcdb2b02ba089d945bd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6a0b6a58cc8a474ab10ae6801d0fb698]), partition=
I20240502 08:18:11.019172 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad09fc1b096c4bcdb2b02ba089d945bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.019124 30401 tablet_service.cc:1467] Processing CreateTablet for tablet ad09fc1b096c4bcdb2b02ba089d945bd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6a0b6a58cc8a474ab10ae6801d0fb698]), partition=
I20240502 08:18:11.019490 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad09fc1b096c4bcdb2b02ba089d945bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.023262 623 consensus_queue.cc:1035] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.025385 30268 tablet_service.cc:1467] Processing CreateTablet for tablet ad09fc1b096c4bcdb2b02ba089d945bd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6a0b6a58cc8a474ab10ae6801d0fb698]), partition=
I20240502 08:18:11.025810 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad09fc1b096c4bcdb2b02ba089d945bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.036521 690 tablet_bootstrap.cc:492] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:11.037986 690 tablet_bootstrap.cc:654] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.039574 691 tablet_bootstrap.cc:492] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:11.042802 690 tablet_bootstrap.cc:492] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:11.042876 691 tablet_bootstrap.cc:654] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.043586 690 ts_tablet_manager.cc:1367] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240502 08:18:11.044462 689 tablet_bootstrap.cc:492] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:11.044941 690 raft_consensus.cc:348] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:11.045167 690 raft_consensus.cc:374] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.045274 690 raft_consensus.cc:729] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.045490 690 consensus_queue.cc:260] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:11.046093 690 ts_tablet_manager.cc:1398] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:11.046674 525 consensus_queue.cc:1035] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.046917 689 tablet_bootstrap.cc:654] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.048522 691 tablet_bootstrap.cc:492] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:11.048723 689 tablet_bootstrap.cc:492] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:11.048882 689 ts_tablet_manager.cc:1367] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:18:11.049589 689 raft_consensus.cc:348] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:11.049844 689 raft_consensus.cc:374] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.049883 691 ts_tablet_manager.cc:1367] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 08:18:11.049965 689 raft_consensus.cc:729] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.050253 689 consensus_queue.cc:260] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:11.050789 689 ts_tablet_manager.cc:1398] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:11.050772 691 raft_consensus.cc:348] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:11.051146 691 raft_consensus.cc:374] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.051298 691 raft_consensus.cc:729] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.051553 691 consensus_queue.cc:260] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:11.052134 691 ts_tablet_manager.cc:1398] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:11.067759 27112 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:11.076431 30268 tablet_service.cc:1467] Processing CreateTablet for tablet f1f416f090284b8d92d53662cc7a3b97 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5ebae50d574942c6976b1ce3e50307d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.076526 30401 tablet_service.cc:1467] Processing CreateTablet for tablet f1f416f090284b8d92d53662cc7a3b97 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5ebae50d574942c6976b1ce3e50307d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.076844 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f416f090284b8d92d53662cc7a3b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.076834 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f416f090284b8d92d53662cc7a3b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.076892 30270 tablet_service.cc:1467] Processing CreateTablet for tablet ecc002b16f4642159e199298acd38392 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5ebae50d574942c6976b1ce3e50307d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.077913 27370 tablet_service.cc:1467] Processing CreateTablet for tablet f1f416f090284b8d92d53662cc7a3b97 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5ebae50d574942c6976b1ce3e50307d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.078359 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f416f090284b8d92d53662cc7a3b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.081991 30140 tablet_service.cc:1467] Processing CreateTablet for tablet ecc002b16f4642159e199298acd38392 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5ebae50d574942c6976b1ce3e50307d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.082250 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc002b16f4642159e199298acd38392. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.082394 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc002b16f4642159e199298acd38392. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.083328 691 tablet_bootstrap.cc:492] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:11.085182 691 tablet_bootstrap.cc:654] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.090262 554 consensus_queue.cc:1035] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.091845 30007 tablet_service.cc:1467] Processing CreateTablet for tablet ecc002b16f4642159e199298acd38392 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5ebae50d574942c6976b1ce3e50307d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.092407 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc002b16f4642159e199298acd38392. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.093125 689 tablet_bootstrap.cc:492] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:11.095214 30401 tablet_service.cc:1467] Processing CreateTablet for tablet 4be4162202014cea96d03cd6c12f5e83 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5ebae50d574942c6976b1ce3e50307d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.095633 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be4162202014cea96d03cd6c12f5e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.091862 30006 tablet_service.cc:1467] Processing CreateTablet for tablet 4be4162202014cea96d03cd6c12f5e83 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5ebae50d574942c6976b1ce3e50307d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.096632 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be4162202014cea96d03cd6c12f5e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.097469 30138 tablet_service.cc:1467] Processing CreateTablet for tablet 4be4162202014cea96d03cd6c12f5e83 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5ebae50d574942c6976b1ce3e50307d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.097879 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be4162202014cea96d03cd6c12f5e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.101087 689 tablet_bootstrap.cc:654] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.109095 689 tablet_bootstrap.cc:492] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:11.109294 700 tablet_bootstrap.cc:492] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:11.109331 689 ts_tablet_manager.cc:1367] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20240502 08:18:11.110150 689 raft_consensus.cc:348] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:11.110450 689 raft_consensus.cc:374] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.110558 689 raft_consensus.cc:729] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.110827 689 consensus_queue.cc:260] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:11.111150 700 tablet_bootstrap.cc:654] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.114300 696 tablet_bootstrap.cc:492] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:11.116133 691 tablet_bootstrap.cc:492] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:11.116335 691 ts_tablet_manager.cc:1367] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.009s
I20240502 08:18:11.117266 691 raft_consensus.cc:348] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:11.117583 689 ts_tablet_manager.cc:1398] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:18:11.117666 691 raft_consensus.cc:374] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.117781 691 raft_consensus.cc:729] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.118110 691 consensus_queue.cc:260] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:11.122152 689 tablet_bootstrap.cc:492] T 4be4162202014cea96d03cd6c12f5e83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:11.123905 696 tablet_bootstrap.cc:654] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.123953 691 ts_tablet_manager.cc:1398] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:18:11.124099 689 tablet_bootstrap.cc:654] T 4be4162202014cea96d03cd6c12f5e83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.134922 691 tablet_bootstrap.cc:492] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:11.137475 691 tablet_bootstrap.cc:654] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.155326 691 tablet_bootstrap.cc:492] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:11.155493 691 ts_tablet_manager.cc:1367] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.001s
I20240502 08:18:11.157284 691 raft_consensus.cc:348] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:11.157568 691 raft_consensus.cc:374] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.157670 691 raft_consensus.cc:729] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.157923 691 consensus_queue.cc:260] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:11.172633 691 ts_tablet_manager.cc:1398] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 08:18:11.174270 708 tablet_bootstrap.cc:492] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:11.181823 696 tablet_bootstrap.cc:492] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:18:11.182077 696 ts_tablet_manager.cc:1367] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.068s user 0.010s sys 0.016s
I20240502 08:18:11.182579 702 consensus_queue.cc:1035] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.183053 696 raft_consensus.cc:348] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:11.183349 696 raft_consensus.cc:374] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.183439 696 raft_consensus.cc:729] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.183703 696 consensus_queue.cc:260] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:11.189131 700 tablet_bootstrap.cc:492] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:11.189383 700 ts_tablet_manager.cc:1367] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.080s user 0.009s sys 0.000s
I20240502 08:18:11.189527 708 tablet_bootstrap.cc:654] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.190577 700 raft_consensus.cc:348] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:11.191463 700 raft_consensus.cc:374] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.192894 700 raft_consensus.cc:729] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.193197 700 consensus_queue.cc:260] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:11.184480 696 ts_tablet_manager.cc:1398] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:11.196048 700 ts_tablet_manager.cc:1398] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:18:11.198827 700 tablet_bootstrap.cc:492] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:11.200526 700 tablet_bootstrap.cc:654] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.218961 616 raft_consensus.cc:482] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.219110 616 raft_consensus.cc:504] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:11.219919 616 leader_election.cc:288] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:11.220455 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad09fc1b096c4bcdb2b02ba089d945bd" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:11.220695 30289 raft_consensus.cc:2454] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:11.221438 30355 leader_election.cc:302] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:18:11.222460 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad09fc1b096c4bcdb2b02ba089d945bd" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:11.222698 27650 raft_consensus.cc:2454] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:11.223829 616 raft_consensus.cc:2781] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.223975 616 raft_consensus.cc:482] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.224089 616 raft_consensus.cc:3037] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.227797 616 raft_consensus.cc:504] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:11.228955 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad09fc1b096c4bcdb2b02ba089d945bd" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:11.229190 27650 raft_consensus.cc:3037] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.232774 27650 raft_consensus.cc:2454] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:11.233942 689 tablet_bootstrap.cc:492] T 4be4162202014cea96d03cd6c12f5e83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:11.234233 689 ts_tablet_manager.cc:1367] T 4be4162202014cea96d03cd6c12f5e83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.112s user 0.016s sys 0.047s
I20240502 08:18:11.234589 30358 leader_election.cc:302] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:11.238855 632 raft_consensus.cc:2781] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.239027 632 raft_consensus.cc:686] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:18:11.239352 632 consensus_queue.cc:237] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:11.239519 616 leader_election.cc:288] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:11.241241 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad09fc1b096c4bcdb2b02ba089d945bd" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:11.241469 30289 raft_consensus.cc:3037] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.243186 689 raft_consensus.cc:348] T 4be4162202014cea96d03cd6c12f5e83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:11.243415 689 raft_consensus.cc:374] T 4be4162202014cea96d03cd6c12f5e83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.243549 689 raft_consensus.cc:729] T 4be4162202014cea96d03cd6c12f5e83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.243505 700 tablet_bootstrap.cc:492] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:11.243741 700 ts_tablet_manager.cc:1367] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.006s
I20240502 08:18:11.243777 689 consensus_queue.cc:260] T 4be4162202014cea96d03cd6c12f5e83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:11.244319 30289 raft_consensus.cc:2454] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:11.244413 689 ts_tablet_manager.cc:1398] T 4be4162202014cea96d03cd6c12f5e83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:11.244660 700 raft_consensus.cc:348] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:11.244900 700 raft_consensus.cc:374] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.245033 700 raft_consensus.cc:729] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.245328 700 consensus_queue.cc:260] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:11.250917 700 ts_tablet_manager.cc:1398] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 08:18:11.251920 525 raft_consensus.cc:482] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.252045 525 raft_consensus.cc:504] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:11.252610 525 leader_election.cc:288] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:11.253157 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f416f090284b8d92d53662cc7a3b97" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:11.253362 30421 raft_consensus.cc:2454] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:11.253902 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f416f090284b8d92d53662cc7a3b97" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:18:11.254163 27388 raft_consensus.cc:2454] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:11.254703 30223 leader_election.cc:302] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:18:11.255757 525 raft_consensus.cc:2781] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.255877 525 raft_consensus.cc:482] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.255954 525 raft_consensus.cc:3037] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.257306 708 tablet_bootstrap.cc:492] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:11.257442 708 ts_tablet_manager.cc:1367] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.083s user 0.012s sys 0.024s
I20240502 08:18:11.258141 708 raft_consensus.cc:348] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:11.258466 708 raft_consensus.cc:374] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.258634 708 raft_consensus.cc:729] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.258816 525 raft_consensus.cc:504] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:11.258988 708 consensus_queue.cc:260] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:11.261569 708 ts_tablet_manager.cc:1398] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:11.262425 708 tablet_bootstrap.cc:492] T 4be4162202014cea96d03cd6c12f5e83 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:11.265272 708 tablet_bootstrap.cc:654] T 4be4162202014cea96d03cd6c12f5e83 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.267858 707 raft_consensus.cc:482] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.267989 707 raft_consensus.cc:504] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:11.268594 707 leader_election.cc:288] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:11.269263 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be4162202014cea96d03cd6c12f5e83" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:11.269312 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be4162202014cea96d03cd6c12f5e83" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:11.269505 30421 raft_consensus.cc:2454] T 4be4162202014cea96d03cd6c12f5e83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
W20240502 08:18:11.269841 29964 leader_election.cc:341] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:11.269992 29964 leader_election.cc:302] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:11.270546 707 raft_consensus.cc:2781] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.270663 707 raft_consensus.cc:482] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.270746 707 raft_consensus.cc:3037] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.273039 707 raft_consensus.cc:504] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:11.273824 708 tablet_bootstrap.cc:492] T 4be4162202014cea96d03cd6c12f5e83 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:11.273831 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be4162202014cea96d03cd6c12f5e83" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:11.274052 30421 raft_consensus.cc:3037] T 4be4162202014cea96d03cd6c12f5e83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.274108 707 leader_election.cc:288] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:11.273950 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be4162202014cea96d03cd6c12f5e83" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
W20240502 08:18:11.274685 29964 leader_election.cc:341] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:11.275044 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f416f090284b8d92d53662cc7a3b97" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:18:11.275215 708 ts_tablet_manager.cc:1367] T 4be4162202014cea96d03cd6c12f5e83 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.002s
I20240502 08:18:11.275234 27388 raft_consensus.cc:3037] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.275923 708 raft_consensus.cc:348] T 4be4162202014cea96d03cd6c12f5e83 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:11.276098 708 raft_consensus.cc:374] T 4be4162202014cea96d03cd6c12f5e83 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.276160 708 raft_consensus.cc:729] T 4be4162202014cea96d03cd6c12f5e83 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.276348 708 consensus_queue.cc:260] T 4be4162202014cea96d03cd6c12f5e83 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:11.276768 708 ts_tablet_manager.cc:1398] T 4be4162202014cea96d03cd6c12f5e83 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:18:11.276770 30421 raft_consensus.cc:2454] T 4be4162202014cea96d03cd6c12f5e83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:11.277629 29961 leader_election.cc:302] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:11.278481 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f416f090284b8d92d53662cc7a3b97" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:11.278713 30421 raft_consensus.cc:3037] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.281124 30421 raft_consensus.cc:2454] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:11.277825 27388 raft_consensus.cc:2454] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:11.281737 30223 leader_election.cc:302] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:18:11.282104 623 raft_consensus.cc:2781] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.282234 623 raft_consensus.cc:686] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:18:11.282518 623 consensus_queue.cc:237] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:11.286389 27516 raft_consensus.cc:1261] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 2 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:18:11.287079 707 raft_consensus.cc:2781] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.287115 639 consensus_queue.cc:1035] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:11.287226 707 raft_consensus.cc:686] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:18:11.287482 707 consensus_queue.cc:237] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:11.288384 525 leader_election.cc:288] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:11.305339 27116 catalog_manager.cc:5484] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:11.312755 27326 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.3:44469, user_credentials={real_user=slave}} blocked reactor thread for 67729.3us
I20240502 08:18:11.313755 27116 catalog_manager.cc:5484] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7). New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:11.314826 27112 catalog_manager.cc:5484] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:11.343725 30027 raft_consensus.cc:1261] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:18:11.344288 639 consensus_queue.cc:1035] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:11.349148 27257 raft_consensus.cc:1261] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:18:11.352510 727 consensus_queue.cc:1035] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:11.398876 623 raft_consensus.cc:482] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.399060 623 raft_consensus.cc:504] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:11.399853 623 leader_election.cc:288] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:11.400238 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc002b16f4642159e199298acd38392" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:11.400285 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc002b16f4642159e199298acd38392" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:11.400470 30157 raft_consensus.cc:2454] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:11.400480 30027 raft_consensus.cc:2454] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:11.400993 30226 leader_election.cc:302] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:11.401757 623 raft_consensus.cc:2781] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.401847 623 raft_consensus.cc:482] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.401959 623 raft_consensus.cc:3037] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.404546 623 raft_consensus.cc:504] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:11.405186 623 leader_election.cc:288] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:11.405570 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc002b16f4642159e199298acd38392" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:11.405658 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc002b16f4642159e199298acd38392" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:11.405763 30157 raft_consensus.cc:3037] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.405840 30027 raft_consensus.cc:3037] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.408601 30157 raft_consensus.cc:2454] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:11.408944 30226 leader_election.cc:302] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:11.409085 30027 raft_consensus.cc:2454] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:11.409173 623 raft_consensus.cc:2781] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.409360 623 raft_consensus.cc:686] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:18:11.409714 623 consensus_queue.cc:237] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:11.413249 27112 catalog_manager.cc:5484] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7). New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:11.462301 27112 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:11.467106 27650 raft_consensus.cc:1261] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.468022 30289 raft_consensus.cc:1261] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.468652 632 consensus_queue.cc:1035] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.475167 27112 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet ad09fc1b096c4bcdb2b02ba089d945bd (table rebalancer_dynamic_table [id=6a0b6a58cc8a474ab10ae6801d0fb698]). Expected version 1 got 0
I20240502 08:18:11.479015 390 consensus_queue.cc:1035] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.479010 732 tablet.cc:1722] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:11.479147 733 tablet.cc:1722] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:11.479534 733 tablet.cc:1620] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:11.479566 732 tablet.cc:1620] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:11.482812 27116 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet ad09fc1b096c4bcdb2b02ba089d945bd (table rebalancer_dynamic_table [id=6a0b6a58cc8a474ab10ae6801d0fb698]). Expected version 1 got 0
I20240502 08:18:11.491204 27112 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=6a0b6a58cc8a474ab10ae6801d0fb698] alter complete (version 1)
I20240502 08:18:11.496889 735 tablet.cc:1722] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:11.497308 735 tablet.cc:1620] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442: MemRowSet was empty: no flush needed.
I20240502 08:18:11.499905 27112 catalog_manager.cc:5576] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309) does not have the latest schema for tablet ad09fc1b096c4bcdb2b02ba089d945bd (table rebalancer_dynamic_table [id=6a0b6a58cc8a474ab10ae6801d0fb698]). Expected version 1 got 0
I20240502 08:18:11.595227 27112 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:11.602844 27112 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet ad09fc1b096c4bcdb2b02ba089d945bd (table rebalancer_dynamic_table [id=6a0b6a58cc8a474ab10ae6801d0fb698]). Expected version 2 got 1
I20240502 08:18:11.604586 733 tablet.cc:1722] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:11.604475 732 tablet.cc:1722] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:11.605044 733 tablet.cc:1620] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:11.605252 732 tablet.cc:1620] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:11.605783 735 tablet.cc:1722] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:11.606173 735 tablet.cc:1620] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442: MemRowSet was empty: no flush needed.
I20240502 08:18:11.608544 300 raft_consensus.cc:482] T 4be4162202014cea96d03cd6c12f5e83 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.608714 300 raft_consensus.cc:504] T 4be4162202014cea96d03cd6c12f5e83 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:11.609402 300 leader_election.cc:288] T 4be4162202014cea96d03cd6c12f5e83 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:11.609978 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be4162202014cea96d03cd6c12f5e83" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:11.610832 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be4162202014cea96d03cd6c12f5e83" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:11.611088 30421 raft_consensus.cc:2379] T 4be4162202014cea96d03cd6c12f5e83 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84cda20738d2452da166e31fc6da57a6 in current term 1: Already voted for candidate 1f0bf4bc3c5146539146b825b048bff3 in this term.
I20240502 08:18:11.613567 30093 leader_election.cc:302] T 4be4162202014cea96d03cd6c12f5e83 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 1f0bf4bc3c5146539146b825b048bff3, 31a1e33dcdbf42f58ea72ce1c0afb0ad
I20240502 08:18:11.613965 300 raft_consensus.cc:3037] T 4be4162202014cea96d03cd6c12f5e83 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.616503 300 raft_consensus.cc:2726] T 4be4162202014cea96d03cd6c12f5e83 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:11.617452 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=6a0b6a58cc8a474ab10ae6801d0fb698] alter complete (version 2)
I20240502 08:18:11.691289 711 consensus_queue.cc:1035] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.696592 554 consensus_queue.cc:1035] T 4be4162202014cea96d03cd6c12f5e83 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.716421 525 consensus_queue.cc:1035] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.720535 626 consensus_queue.cc:1035] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.721897 27112 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:11.734405 735 tablet.cc:1722] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:11.736083 733 tablet.cc:1722] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:11.747155 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=6a0b6a58cc8a474ab10ae6801d0fb698] alter complete (version 3)
I20240502 08:18:11.749795 732 tablet.cc:1722] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:11.850569 27116 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:11.850800 27116 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:11.853480 27116 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet ad09fc1b096c4bcdb2b02ba089d945bd
I20240502 08:18:11.854198 27631 tablet_service.cc:1514] Processing DeleteTablet for tablet ad09fc1b096c4bcdb2b02ba089d945bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:11 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:18:11.854604 30401 tablet_service.cc:1514] Processing DeleteTablet for tablet ad09fc1b096c4bcdb2b02ba089d945bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:11 UTC) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:11.854732 754 tablet_replica.cc:330] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:18:11.854965 754 raft_consensus.cc:2229] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:11.854984 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet ad09fc1b096c4bcdb2b02ba089d945bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:11 UTC) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:11.855190 754 raft_consensus.cc:2258] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:11.855284 755 tablet_replica.cc:330] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:11.855557 755 raft_consensus.cc:2229] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:11.855962 755 raft_consensus.cc:2258] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:11.856072 756 tablet_replica.cc:330] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:11.856163 754 ts_tablet_manager.cc:1875] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:11.856622 755 ts_tablet_manager.cc:1875] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:11.856640 756 raft_consensus.cc:2229] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:11.856846 756 raft_consensus.cc:2258] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:11.857601 756 ts_tablet_manager.cc:1875] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:11.861575 755 ts_tablet_manager.cc:1888] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:11.861709 755 log.cc:1196] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/ad09fc1b096c4bcdb2b02ba089d945bd
I20240502 08:18:11.862167 755 ts_tablet_manager.cc:1909] T ad09fc1b096c4bcdb2b02ba089d945bd P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting consensus metadata
I20240502 08:18:11.862663 754 ts_tablet_manager.cc:1888] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:11.862766 754 log.cc:1196] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/ad09fc1b096c4bcdb2b02ba089d945bd
I20240502 08:18:11.863073 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet ad09fc1b096c4bcdb2b02ba089d945bd (table rebalancer_dynamic_table_new_name [id=6a0b6a58cc8a474ab10ae6801d0fb698]) successfully deleted
I20240502 08:18:11.863579 754 ts_tablet_manager.cc:1909] T ad09fc1b096c4bcdb2b02ba089d945bd P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:18:11.863703 756 ts_tablet_manager.cc:1888] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:11.863816 756 log.cc:1196] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/ad09fc1b096c4bcdb2b02ba089d945bd
I20240502 08:18:11.864236 756 ts_tablet_manager.cc:1909] T ad09fc1b096c4bcdb2b02ba089d945bd P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting consensus metadata
I20240502 08:18:11.864567 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet ad09fc1b096c4bcdb2b02ba089d945bd (table rebalancer_dynamic_table_new_name [id=6a0b6a58cc8a474ab10ae6801d0fb698]) successfully deleted
I20240502 08:18:11.865361 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet ad09fc1b096c4bcdb2b02ba089d945bd (table rebalancer_dynamic_table_new_name [id=6a0b6a58cc8a474ab10ae6801d0fb698]) successfully deleted
I20240502 08:18:11.870010 745 consensus_queue.cc:1035] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.877125 626 consensus_queue.cc:1035] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.903124 27650 consensus_queue.cc:237] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:11.904709 27257 raft_consensus.cc:1261] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:18:11.904805 30027 raft_consensus.cc:1261] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:18:11.905303 727 consensus_queue.cc:1035] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:11.905637 31753 consensus_queue.cc:1035] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:11.907290 727 raft_consensus.cc:2932] T 9725e08a6c5d4410b8d944ab154ec1eb P bf8ac71b15f740a0a928911360773442 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:11.908247 27257 raft_consensus.cc:2932] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:11.910774 30028 raft_consensus.cc:2932] T 9725e08a6c5d4410b8d944ab154ec1eb P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:11.913412 27110 catalog_manager.cc:5484] T 9725e08a6c5d4410b8d944ab154ec1eb P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 5 to 7, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 2 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:11.915656 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9725e08a6c5d4410b8d944ab154ec1eb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:11.923322 27110 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:11.926944 27498 tablet_service.cc:1514] Processing DeleteTablet for tablet 9725e08a6c5d4410b8d944ab154ec1eb with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:11.928408 761 tablet_replica.cc:330] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:11.928731 761 raft_consensus.cc:2229] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:11.928907 761 raft_consensus.cc:2258] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:11.930701 761 ts_tablet_manager.cc:1875] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:11.935487 30270 tablet_service.cc:1467] Processing CreateTablet for tablet 4c78e068ec3a451c892d58174b3f837c (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=0f9f00e51ec94f5f9d43f80dbb78ff7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.935958 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c78e068ec3a451c892d58174b3f837c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.936831 27370 tablet_service.cc:1467] Processing CreateTablet for tablet 95fc757372c84ca9928f9ae8133b33f7 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=0f9f00e51ec94f5f9d43f80dbb78ff7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.937144 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fc757372c84ca9928f9ae8133b33f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.938448 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 95fc757372c84ca9928f9ae8133b33f7 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=0f9f00e51ec94f5f9d43f80dbb78ff7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.938881 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fc757372c84ca9928f9ae8133b33f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.940976 761 ts_tablet_manager.cc:1888] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:18:11.941094 761 log.cc:1196] T 9725e08a6c5d4410b8d944ab154ec1eb P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/9725e08a6c5d4410b8d944ab154ec1eb
I20240502 08:18:11.942027 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 9725e08a6c5d4410b8d944ab154ec1eb (table rebalance_test_table_13 [id=28a67fad5bff471981e8cdf2b2a2b34a]) successfully deleted
I20240502 08:18:11.943527 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 4c78e068ec3a451c892d58174b3f837c (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=0f9f00e51ec94f5f9d43f80dbb78ff7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.943554 27629 tablet_service.cc:1467] Processing CreateTablet for tablet 56542089a8e14afb9ba3d9e7cbbbcdf6 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=0f9f00e51ec94f5f9d43f80dbb78ff7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.944033 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c78e068ec3a451c892d58174b3f837c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.944260 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56542089a8e14afb9ba3d9e7cbbbcdf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.951313 30007 tablet_service.cc:1467] Processing CreateTablet for tablet 56542089a8e14afb9ba3d9e7cbbbcdf6 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=0f9f00e51ec94f5f9d43f80dbb78ff7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.951716 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56542089a8e14afb9ba3d9e7cbbbcdf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.951719 30400 tablet_service.cc:1467] Processing CreateTablet for tablet 95fc757372c84ca9928f9ae8133b33f7 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=0f9f00e51ec94f5f9d43f80dbb78ff7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.952103 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fc757372c84ca9928f9ae8133b33f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.954947 763 tablet_bootstrap.cc:492] T 95fc757372c84ca9928f9ae8133b33f7 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:11.955984 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 56542089a8e14afb9ba3d9e7cbbbcdf6 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=0f9f00e51ec94f5f9d43f80dbb78ff7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.956394 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56542089a8e14afb9ba3d9e7cbbbcdf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.961498 30401 tablet_service.cc:1467] Processing CreateTablet for tablet 4c78e068ec3a451c892d58174b3f837c (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=0f9f00e51ec94f5f9d43f80dbb78ff7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.961970 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c78e068ec3a451c892d58174b3f837c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.963855 764 tablet_bootstrap.cc:492] T 95fc757372c84ca9928f9ae8133b33f7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:11.963882 765 tablet_bootstrap.cc:492] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:11.965198 765 tablet_bootstrap.cc:654] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.965404 763 tablet_bootstrap.cc:654] T 95fc757372c84ca9928f9ae8133b33f7 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.967455 764 tablet_bootstrap.cc:654] T 95fc757372c84ca9928f9ae8133b33f7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.975112 764 tablet_bootstrap.cc:492] T 95fc757372c84ca9928f9ae8133b33f7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:11.975369 764 ts_tablet_manager.cc:1367] T 95fc757372c84ca9928f9ae8133b33f7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.003s
I20240502 08:18:11.976069 764 raft_consensus.cc:348] T 95fc757372c84ca9928f9ae8133b33f7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:11.976393 764 raft_consensus.cc:374] T 95fc757372c84ca9928f9ae8133b33f7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.976536 764 raft_consensus.cc:729] T 95fc757372c84ca9928f9ae8133b33f7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.977824 764 consensus_queue.cc:260] T 95fc757372c84ca9928f9ae8133b33f7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:11.978538 764 ts_tablet_manager.cc:1398] T 95fc757372c84ca9928f9ae8133b33f7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:11.979293 763 tablet_bootstrap.cc:492] T 95fc757372c84ca9928f9ae8133b33f7 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:18:11.979473 763 ts_tablet_manager.cc:1367] T 95fc757372c84ca9928f9ae8133b33f7 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20240502 08:18:11.980161 762 tablet_bootstrap.cc:492] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:11.980324 763 raft_consensus.cc:348] T 95fc757372c84ca9928f9ae8133b33f7 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:11.980552 763 raft_consensus.cc:374] T 95fc757372c84ca9928f9ae8133b33f7 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.980644 763 raft_consensus.cc:729] T 95fc757372c84ca9928f9ae8133b33f7 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.980993 763 consensus_queue.cc:260] T 95fc757372c84ca9928f9ae8133b33f7 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:11.981523 763 ts_tablet_manager.cc:1398] T 95fc757372c84ca9928f9ae8133b33f7 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:11.981510 27112 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:11.983103 762 tablet_bootstrap.cc:654] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.990227 762 tablet_bootstrap.cc:492] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:11.990375 762 ts_tablet_manager.cc:1367] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20240502 08:18:11.991101 762 raft_consensus.cc:348] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:11.991356 762 raft_consensus.cc:374] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.991500 762 raft_consensus.cc:729] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.991739 762 consensus_queue.cc:260] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:11.994498 766 tablet_bootstrap.cc:492] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:11.994256 762 ts_tablet_manager.cc:1398] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:18:11.994817 762 tablet_bootstrap.cc:492] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:11.995699 27629 tablet_service.cc:1467] Processing CreateTablet for tablet 9d4ae1ded24647ff96a67c48c4ee2b1d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ce0784c07b994556b15b79a8b641aad2]), partition=
I20240502 08:18:11.996018 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d4ae1ded24647ff96a67c48c4ee2b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.996023 30270 tablet_service.cc:1467] Processing CreateTablet for tablet 9d4ae1ded24647ff96a67c48c4ee2b1d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ce0784c07b994556b15b79a8b641aad2]), partition=
I20240502 08:18:11.996155 766 tablet_bootstrap.cc:654] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.996301 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d4ae1ded24647ff96a67c48c4ee2b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.998310 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 9d4ae1ded24647ff96a67c48c4ee2b1d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ce0784c07b994556b15b79a8b641aad2]), partition=
I20240502 08:18:11.998685 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d4ae1ded24647ff96a67c48c4ee2b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.003823 765 tablet_bootstrap.cc:492] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:12.004020 765 ts_tablet_manager.cc:1367] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20240502 08:18:12.004748 765 raft_consensus.cc:348] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:12.005000 765 raft_consensus.cc:374] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.005209 765 raft_consensus.cc:729] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.005523 765 consensus_queue.cc:260] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:12.007082 765 ts_tablet_manager.cc:1398] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:12.007375 765 tablet_bootstrap.cc:492] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:12.008903 765 tablet_bootstrap.cc:654] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.009459 770 tablet_bootstrap.cc:492] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:12.011580 770 tablet_bootstrap.cc:654] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.012298 762 tablet_bootstrap.cc:654] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.012573 764 tablet_bootstrap.cc:492] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:12.014331 764 tablet_bootstrap.cc:654] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.018226 766 tablet_bootstrap.cc:492] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:12.018453 766 ts_tablet_manager.cc:1367] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.006s
I20240502 08:18:12.019757 766 raft_consensus.cc:348] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:12.022186 766 raft_consensus.cc:374] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.022359 766 raft_consensus.cc:729] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.023847 764 tablet_bootstrap.cc:492] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:12.024020 764 ts_tablet_manager.cc:1367] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20240502 08:18:12.024298 766 consensus_queue.cc:260] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:12.024724 764 raft_consensus.cc:348] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:12.024958 764 raft_consensus.cc:374] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.025050 764 raft_consensus.cc:729] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.027014 764 consensus_queue.cc:260] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:12.029939 764 ts_tablet_manager.cc:1398] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:18:12.034713 762 tablet_bootstrap.cc:492] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:12.035279 762 ts_tablet_manager.cc:1367] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.001s
I20240502 08:18:12.036859 762 raft_consensus.cc:348] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:12.037648 762 raft_consensus.cc:374] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.038275 762 raft_consensus.cc:729] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.038841 762 consensus_queue.cc:260] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:12.040602 762 ts_tablet_manager.cc:1398] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:12.040848 762 tablet_bootstrap.cc:492] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:12.041568 766 ts_tablet_manager.cc:1398] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.023s user 0.000s sys 0.002s
I20240502 08:18:12.050029 766 tablet_bootstrap.cc:492] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:12.042907 762 tablet_bootstrap.cc:654] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.057431 766 tablet_bootstrap.cc:654] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.065979 766 tablet_bootstrap.cc:492] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:12.066166 766 ts_tablet_manager.cc:1367] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.005s
I20240502 08:18:12.068787 766 raft_consensus.cc:348] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:12.069025 766 raft_consensus.cc:374] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.069120 766 raft_consensus.cc:729] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.071861 766 consensus_queue.cc:260] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:12.072407 766 ts_tablet_manager.cc:1398] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 08:18:12.081487 770 tablet_bootstrap.cc:492] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:12.081673 770 ts_tablet_manager.cc:1367] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.072s user 0.020s sys 0.013s
I20240502 08:18:12.082494 770 raft_consensus.cc:348] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:12.082717 770 raft_consensus.cc:374] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.082816 770 raft_consensus.cc:729] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.083051 770 consensus_queue.cc:260] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:12.083611 770 ts_tablet_manager.cc:1398] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:12.089337 765 tablet_bootstrap.cc:492] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:12.089478 765 ts_tablet_manager.cc:1367] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.082s user 0.006s sys 0.026s
I20240502 08:18:12.090142 765 raft_consensus.cc:348] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:12.090389 765 raft_consensus.cc:374] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.090493 765 raft_consensus.cc:729] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.090737 765 consensus_queue.cc:260] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:12.091297 765 ts_tablet_manager.cc:1398] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:12.091552 765 tablet_bootstrap.cc:492] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:12.092991 765 tablet_bootstrap.cc:654] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.098765 32172 raft_consensus.cc:482] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:12.098968 32172 raft_consensus.cc:504] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:12.099793 32172 leader_election.cc:288] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:12.100841 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4ae1ded24647ff96a67c48c4ee2b1d" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:12.101931 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4ae1ded24647ff96a67c48c4ee2b1d" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
W20240502 08:18:12.102460 27191 leader_election.cc:341] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:18:12.104357 27194 leader_election.cc:341] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:12.104519 27194 leader_election.cc:302] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, bf8ac71b15f740a0a928911360773442
I20240502 08:18:12.104826 32172 raft_consensus.cc:2726] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:12.114269 765 tablet_bootstrap.cc:492] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:12.114480 765 ts_tablet_manager.cc:1367] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.004s
I20240502 08:18:12.115113 762 tablet_bootstrap.cc:492] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:12.115109 765 raft_consensus.cc:348] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:12.115312 765 raft_consensus.cc:374] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.115315 762 ts_tablet_manager.cc:1367] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.075s user 0.000s sys 0.009s
I20240502 08:18:12.115387 765 raft_consensus.cc:729] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.115618 765 consensus_queue.cc:260] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:12.115944 762 raft_consensus.cc:348] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:12.116084 765 ts_tablet_manager.cc:1398] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:18:12.116200 762 raft_consensus.cc:374] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.116345 762 raft_consensus.cc:729] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.116552 762 consensus_queue.cc:260] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:12.117179 762 ts_tablet_manager.cc:1398] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:12.132897 390 raft_consensus.cc:482] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:12.133047 390 raft_consensus.cc:504] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:12.133944 390 leader_election.cc:288] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:12.134325 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c78e068ec3a451c892d58174b3f837c" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:12.134570 27650 raft_consensus.cc:2454] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:12.134903 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c78e068ec3a451c892d58174b3f837c" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:12.135077 30358 leader_election.cc:302] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:12.135169 30289 raft_consensus.cc:2454] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:12.135401 390 raft_consensus.cc:2781] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:12.135524 390 raft_consensus.cc:482] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:12.135618 390 raft_consensus.cc:3037] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.139273 390 raft_consensus.cc:504] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:12.142606 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c78e068ec3a451c892d58174b3f837c" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:12.142835 27650 raft_consensus.cc:3037] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.145040 760 raft_consensus.cc:482] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:12.145195 760 raft_consensus.cc:504] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:12.145977 760 leader_election.cc:288] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:12.146479 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c78e068ec3a451c892d58174b3f837c" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:12.146745 30289 raft_consensus.cc:3037] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.146991 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56542089a8e14afb9ba3d9e7cbbbcdf6" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:12.147313 30288 raft_consensus.cc:2454] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:18:12.148387 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56542089a8e14afb9ba3d9e7cbbbcdf6" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:12.148643 30028 raft_consensus.cc:2454] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:18:12.149165 27584 leader_election.cc:302] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:12.149816 760 raft_consensus.cc:2781] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:12.149884 760 raft_consensus.cc:482] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:12.149962 760 raft_consensus.cc:3037] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.151230 27650 raft_consensus.cc:2454] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:12.151388 30289 raft_consensus.cc:2454] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:12.151728 30355 leader_election.cc:302] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:18:12.152076 616 raft_consensus.cc:2781] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:12.152137 390 leader_election.cc:288] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:12.152222 616 raft_consensus.cc:686] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:18:12.152511 616 consensus_queue.cc:237] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:12.152722 760 raft_consensus.cc:504] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:12.154137 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56542089a8e14afb9ba3d9e7cbbbcdf6" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:12.154368 30028 raft_consensus.cc:3037] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.156764 30028 raft_consensus.cc:2454] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:12.156819 27116 catalog_manager.cc:5484] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:12.157143 27584 leader_election.cc:302] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:12.157377 31753 raft_consensus.cc:2781] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:12.157490 31753 raft_consensus.cc:686] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:18:12.157745 31753 consensus_queue.cc:237] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:12.159909 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56542089a8e14afb9ba3d9e7cbbbcdf6" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:12.160118 30289 raft_consensus.cc:3037] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.157480 760 leader_election.cc:288] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:12.162634 30289 raft_consensus.cc:2454] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:12.163084 27112 catalog_manager.cc:5484] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:12.206894 760 raft_consensus.cc:482] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:12.207062 760 raft_consensus.cc:504] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:12.207857 760 leader_election.cc:288] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:12.209493 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4ae1ded24647ff96a67c48c4ee2b1d" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:12.210281 27257 raft_consensus.cc:2454] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:18:12.210884 27586 leader_election.cc:302] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:12.211123 760 raft_consensus.cc:2781] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:12.211226 760 raft_consensus.cc:482] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:12.211309 760 raft_consensus.cc:3037] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.209834 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4ae1ded24647ff96a67c48c4ee2b1d" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:12.211570 30289 raft_consensus.cc:2454] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:18:12.214534 760 raft_consensus.cc:504] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:12.215410 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4ae1ded24647ff96a67c48c4ee2b1d" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:12.215612 30289 raft_consensus.cc:3037] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.215852 760 leader_election.cc:288] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:12.218411 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4ae1ded24647ff96a67c48c4ee2b1d" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:12.218659 27257 raft_consensus.cc:3037] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.221495 27257 raft_consensus.cc:2454] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:12.222791 27586 leader_election.cc:302] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:12.223042 760 raft_consensus.cc:2781] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:12.223135 760 raft_consensus.cc:686] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:18:12.223363 760 consensus_queue.cc:237] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:12.225312 30289 raft_consensus.cc:2454] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:12.229061 27112 catalog_manager.cc:5484] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:12.265202 616 raft_consensus.cc:482] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:12.265388 616 raft_consensus.cc:504] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:12.266185 616 leader_election.cc:288] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:12.267119 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fc757372c84ca9928f9ae8133b33f7" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:18:12.267319 27388 raft_consensus.cc:2454] T 95fc757372c84ca9928f9ae8133b33f7 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:12.267359 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fc757372c84ca9928f9ae8133b33f7" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:12.267561 27257 raft_consensus.cc:2454] T 95fc757372c84ca9928f9ae8133b33f7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:12.268151 30357 leader_election.cc:302] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:12.268405 616 raft_consensus.cc:2781] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:12.268488 616 raft_consensus.cc:482] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:12.268607 616 raft_consensus.cc:3037] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.271272 616 raft_consensus.cc:504] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:12.271818 616 leader_election.cc:288] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:12.272361 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fc757372c84ca9928f9ae8133b33f7" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:12.272588 27257 raft_consensus.cc:3037] T 95fc757372c84ca9928f9ae8133b33f7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.273150 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fc757372c84ca9928f9ae8133b33f7" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:18:12.273340 27388 raft_consensus.cc:3037] T 95fc757372c84ca9928f9ae8133b33f7 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.275542 27257 raft_consensus.cc:2454] T 95fc757372c84ca9928f9ae8133b33f7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:12.276086 27388 raft_consensus.cc:2454] T 95fc757372c84ca9928f9ae8133b33f7 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:12.276036 30357 leader_election.cc:302] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters:
I20240502 08:18:12.276437 616 raft_consensus.cc:2781] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:12.276690 616 raft_consensus.cc:686] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:18:12.277067 616 consensus_queue.cc:237] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:12.302982 27111 catalog_manager.cc:5484] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:12.371688 27113 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:12.379209 27257 raft_consensus.cc:1261] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:12.380527 30288 raft_consensus.cc:1261] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:12.380895 760 consensus_queue.cc:1035] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.381117 31753 consensus_queue.cc:1035] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.427021 27115 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet 9d4ae1ded24647ff96a67c48c4ee2b1d (table rebalancer_dynamic_table [id=ce0784c07b994556b15b79a8b641aad2]). Expected version 1 got 0
I20240502 08:18:12.442538 833 tablet.cc:1722] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:12.443046 833 tablet.cc:1620] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442: MemRowSet was empty: no flush needed.
I20240502 08:18:12.445896 27109 catalog_manager.cc:5576] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309) does not have the latest schema for tablet 9d4ae1ded24647ff96a67c48c4ee2b1d (table rebalancer_dynamic_table [id=ce0784c07b994556b15b79a8b641aad2]). Expected version 1 got 0
I20240502 08:18:12.457607 27113 catalog_manager.cc:5576] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505) does not have the latest schema for tablet 9d4ae1ded24647ff96a67c48c4ee2b1d (table rebalancer_dynamic_table [id=ce0784c07b994556b15b79a8b641aad2]). Expected version 1 got 0
I20240502 08:18:12.472961 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=ce0784c07b994556b15b79a8b641aad2] alter complete (version 1)
I20240502 08:18:12.476769 835 tablet.cc:1722] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:12.477754 835 tablet.cc:1620] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: MemRowSet was empty: no flush needed.
I20240502 08:18:12.478717 836 tablet.cc:1722] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:12.479290 836 tablet.cc:1620] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:12.481402 27113 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet 9d4ae1ded24647ff96a67c48c4ee2b1d (table rebalancer_dynamic_table [id=ce0784c07b994556b15b79a8b641aad2]). Expected version 1 got 0
I20240502 08:18:12.510306 27238 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:12.531563 616 consensus_queue.cc:1035] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.537180 760 consensus_queue.cc:1035] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.544131 727 consensus_queue.cc:1035] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.586122 27115 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:12.593616 390 consensus_queue.cc:1035] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.593945 30007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:12.598130 833 tablet.cc:1722] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:12.598695 833 tablet.cc:1620] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442: MemRowSet was empty: no flush needed.
I20240502 08:18:12.602139 835 tablet.cc:1722] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:12.602666 835 tablet.cc:1620] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: MemRowSet was empty: no flush needed.
I20240502 08:18:12.608700 836 tablet.cc:1722] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:12.609265 836 tablet.cc:1620] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:12.616662 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=ce0784c07b994556b15b79a8b641aad2] alter complete (version 2)
I20240502 08:18:12.642014 30400 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:18:12.642895 30093 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.8:41619, user_credentials={real_user=slave}} blocked reactor thread for 65818.3us
I20240502 08:18:12.649889 27629 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:12.666878 30270 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:18:12.680773 27498 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:12.681208 30140 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:12.681361 390 consensus_queue.cc:1035] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.687263 616 consensus_queue.cc:1035] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.695632 27370 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:12.719375 27115 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:12.730124 833 tablet.cc:1722] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:12.734612 836 tablet.cc:1722] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:12.752673 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=ce0784c07b994556b15b79a8b641aad2] alter complete (version 3)
I20240502 08:18:12.764187 835 tablet.cc:1722] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:12.855549 27115 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:12.855803 27115 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:12.859122 27115 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 9d4ae1ded24647ff96a67c48c4ee2b1d
I20240502 08:18:12.859865 27629 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d4ae1ded24647ff96a67c48c4ee2b1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:12 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:18:12.860280 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d4ae1ded24647ff96a67c48c4ee2b1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:12 UTC) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:12.860522 868 tablet_replica.cc:330] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:18:12.860587 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d4ae1ded24647ff96a67c48c4ee2b1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:12 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:18:12.860692 868 raft_consensus.cc:2229] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:12.860886 869 tablet_replica.cc:330] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:12.860988 868 raft_consensus.cc:2258] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:12.861124 869 raft_consensus.cc:2229] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:12.861371 869 raft_consensus.cc:2258] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:12.862924 869 ts_tablet_manager.cc:1875] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:12.866541 868 ts_tablet_manager.cc:1875] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:12.868752 27115 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:12.871532 869 ts_tablet_manager.cc:1888] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:12.871701 869 log.cc:1196] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/9d4ae1ded24647ff96a67c48c4ee2b1d
I20240502 08:18:12.872241 869 ts_tablet_manager.cc:1909] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting consensus metadata
I20240502 08:18:12.874370 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet 9d4ae1ded24647ff96a67c48c4ee2b1d (table rebalancer_dynamic_table_new_name [id=ce0784c07b994556b15b79a8b641aad2]) successfully deleted
I20240502 08:18:12.880365 27370 tablet_service.cc:1467] Processing CreateTablet for tablet 4165b86bc05e47f6a9803fb45ca01569 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=beadc4ad4dcd49e19ca5461ae00bb35f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.880745 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4165b86bc05e47f6a9803fb45ca01569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.880792 27498 tablet_service.cc:1467] Processing CreateTablet for tablet 4165b86bc05e47f6a9803fb45ca01569 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=beadc4ad4dcd49e19ca5461ae00bb35f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.881139 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4165b86bc05e47f6a9803fb45ca01569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.881372 30140 tablet_service.cc:1467] Processing CreateTablet for tablet 4165b86bc05e47f6a9803fb45ca01569 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=beadc4ad4dcd49e19ca5461ae00bb35f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.881742 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4165b86bc05e47f6a9803fb45ca01569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.882366 27368 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed6b8adc62f471696cbf0330524e353 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=beadc4ad4dcd49e19ca5461ae00bb35f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.882668 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed6b8adc62f471696cbf0330524e353. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.888574 27500 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed6b8adc62f471696cbf0330524e353 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=beadc4ad4dcd49e19ca5461ae00bb35f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.888963 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed6b8adc62f471696cbf0330524e353. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.897697 30138 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed6b8adc62f471696cbf0330524e353 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=beadc4ad4dcd49e19ca5461ae00bb35f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.898115 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed6b8adc62f471696cbf0330524e353. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.900848 871 tablet_bootstrap.cc:492] T 4165b86bc05e47f6a9803fb45ca01569 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:12.902477 872 tablet_bootstrap.cc:492] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:12.904592 872 tablet_bootstrap.cc:654] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.904197 871 tablet_bootstrap.cc:654] T 4165b86bc05e47f6a9803fb45ca01569 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.908955 30007 tablet_service.cc:1467] Processing CreateTablet for tablet b8b7b807126c40939ef272fd355b54e4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=beadc4ad4dcd49e19ca5461ae00bb35f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.909391 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b7b807126c40939ef272fd355b54e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.913513 872 tablet_bootstrap.cc:492] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:18:12.913729 872 ts_tablet_manager.cc:1367] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 08:18:12.914891 872 raft_consensus.cc:348] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:12.915647 872 raft_consensus.cc:374] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.915774 872 raft_consensus.cc:729] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.916054 872 consensus_queue.cc:260] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:12.916708 872 ts_tablet_manager.cc:1398] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:12.917016 872 tablet_bootstrap.cc:492] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:12.918578 872 tablet_bootstrap.cc:654] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.921573 871 tablet_bootstrap.cc:492] T 4165b86bc05e47f6a9803fb45ca01569 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:12.921772 871 ts_tablet_manager.cc:1367] T 4165b86bc05e47f6a9803fb45ca01569 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.004s
I20240502 08:18:12.923293 877 tablet_bootstrap.cc:492] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:12.925647 877 tablet_bootstrap.cc:654] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.926527 30270 tablet_service.cc:1467] Processing CreateTablet for tablet b8b7b807126c40939ef272fd355b54e4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=beadc4ad4dcd49e19ca5461ae00bb35f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.927058 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b7b807126c40939ef272fd355b54e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.927369 871 raft_consensus.cc:348] T 4165b86bc05e47f6a9803fb45ca01569 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:12.927595 871 raft_consensus.cc:374] T 4165b86bc05e47f6a9803fb45ca01569 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.927665 871 raft_consensus.cc:729] T 4165b86bc05e47f6a9803fb45ca01569 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.925690 30139 tablet_service.cc:1467] Processing CreateTablet for tablet b8b7b807126c40939ef272fd355b54e4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=beadc4ad4dcd49e19ca5461ae00bb35f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.927891 871 consensus_queue.cc:260] T 4165b86bc05e47f6a9803fb45ca01569 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:12.928123 30139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b7b807126c40939ef272fd355b54e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.928565 871 ts_tablet_manager.cc:1398] T 4165b86bc05e47f6a9803fb45ca01569 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 08:18:12.931704 871 tablet_bootstrap.cc:492] T 0ed6b8adc62f471696cbf0330524e353 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:12.935213 871 tablet_bootstrap.cc:654] T 0ed6b8adc62f471696cbf0330524e353 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.948535 868 ts_tablet_manager.cc:1888] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:12.949518 870 tablet_replica.cc:330] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:18:12.949765 870 raft_consensus.cc:2229] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:12.950048 870 raft_consensus.cc:2258] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:12.951872 878 tablet_bootstrap.cc:492] T b8b7b807126c40939ef272fd355b54e4 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:12.952370 877 tablet_bootstrap.cc:492] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:12.952463 870 ts_tablet_manager.cc:1875] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:12.952987 877 ts_tablet_manager.cc:1367] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.012s
I20240502 08:18:12.954123 878 tablet_bootstrap.cc:654] T b8b7b807126c40939ef272fd355b54e4 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.954170 877 raft_consensus.cc:348] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:12.955516 877 raft_consensus.cc:374] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.955838 877 raft_consensus.cc:729] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.956496 877 consensus_queue.cc:260] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:12.957827 877 ts_tablet_manager.cc:1398] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:12.958647 877 tablet_bootstrap.cc:492] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:12.959415 876 tablet_bootstrap.cc:492] T b8b7b807126c40939ef272fd355b54e4 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:12.961378 27115 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:12.961851 871 tablet_bootstrap.cc:492] T 0ed6b8adc62f471696cbf0330524e353 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:12.962195 877 tablet_bootstrap.cc:654] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.962209 871 ts_tablet_manager.cc:1367] T 0ed6b8adc62f471696cbf0330524e353 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.005s
I20240502 08:18:12.963492 871 raft_consensus.cc:348] T 0ed6b8adc62f471696cbf0330524e353 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:12.964141 876 tablet_bootstrap.cc:654] T b8b7b807126c40939ef272fd355b54e4 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.964269 871 raft_consensus.cc:374] T 0ed6b8adc62f471696cbf0330524e353 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.964404 871 raft_consensus.cc:729] T 0ed6b8adc62f471696cbf0330524e353 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.964704 871 consensus_queue.cc:260] T 0ed6b8adc62f471696cbf0330524e353 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:12.967605 871 ts_tablet_manager.cc:1398] T 0ed6b8adc62f471696cbf0330524e353 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:12.967767 868 log.cc:1196] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/9d4ae1ded24647ff96a67c48c4ee2b1d
I20240502 08:18:12.968247 868 ts_tablet_manager.cc:1909] T 9d4ae1ded24647ff96a67c48c4ee2b1d P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:18:12.973778 30270 tablet_service.cc:1467] Processing CreateTablet for tablet a896dba4b8b34b6da87cf9dd401dbf5d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0b8cec7dd5e64eafaf141c89271018ab]), partition=
I20240502 08:18:12.973821 27238 tablet_service.cc:1467] Processing CreateTablet for tablet a896dba4b8b34b6da87cf9dd401dbf5d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0b8cec7dd5e64eafaf141c89271018ab]), partition=
I20240502 08:18:12.973894 30400 tablet_service.cc:1467] Processing CreateTablet for tablet a896dba4b8b34b6da87cf9dd401dbf5d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0b8cec7dd5e64eafaf141c89271018ab]), partition=
I20240502 08:18:12.974179 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a896dba4b8b34b6da87cf9dd401dbf5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.974179 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a896dba4b8b34b6da87cf9dd401dbf5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.974272 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a896dba4b8b34b6da87cf9dd401dbf5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.974411 877 tablet_bootstrap.cc:492] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:12.974556 877 ts_tablet_manager.cc:1367] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.005s
I20240502 08:18:12.975276 877 raft_consensus.cc:348] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:12.975467 877 raft_consensus.cc:374] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.975528 877 raft_consensus.cc:729] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.977335 877 consensus_queue.cc:260] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:12.978421 877 ts_tablet_manager.cc:1398] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:12.986675 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 9d4ae1ded24647ff96a67c48c4ee2b1d (table rebalancer_dynamic_table_new_name [id=ce0784c07b994556b15b79a8b641aad2]) successfully deleted
I20240502 08:18:12.990409 877 tablet_bootstrap.cc:492] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:12.994359 870 ts_tablet_manager.cc:1888] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:12.996246 877 tablet_bootstrap.cc:654] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.997018 870 log.cc:1196] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/9d4ae1ded24647ff96a67c48c4ee2b1d
I20240502 08:18:12.996614 885 tablet_bootstrap.cc:492] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:12.999051 870 ts_tablet_manager.cc:1909] T 9d4ae1ded24647ff96a67c48c4ee2b1d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:18:13.002938 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet 9d4ae1ded24647ff96a67c48c4ee2b1d (table rebalancer_dynamic_table_new_name [id=ce0784c07b994556b15b79a8b641aad2]) successfully deleted
I20240502 08:18:13.004464 872 tablet_bootstrap.cc:492] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:18:13.005102 872 ts_tablet_manager.cc:1367] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.088s user 0.017s sys 0.031s
I20240502 08:18:13.005892 884 tablet_bootstrap.cc:492] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:13.006446 872 raft_consensus.cc:348] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:13.006665 885 tablet_bootstrap.cc:654] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.006716 872 raft_consensus.cc:374] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.006916 872 raft_consensus.cc:729] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.007369 872 consensus_queue.cc:260] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:13.008160 872 ts_tablet_manager.cc:1398] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:18:13.008594 876 tablet_bootstrap.cc:492] T b8b7b807126c40939ef272fd355b54e4 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:13.008762 876 ts_tablet_manager.cc:1367] T b8b7b807126c40939ef272fd355b54e4 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.000s
I20240502 08:18:13.009606 876 raft_consensus.cc:348] T b8b7b807126c40939ef272fd355b54e4 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:13.009835 876 raft_consensus.cc:374] T b8b7b807126c40939ef272fd355b54e4 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.009958 876 raft_consensus.cc:729] T b8b7b807126c40939ef272fd355b54e4 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.010233 877 tablet_bootstrap.cc:492] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:13.010375 877 ts_tablet_manager.cc:1367] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.004s
I20240502 08:18:13.010228 876 consensus_queue.cc:260] T b8b7b807126c40939ef272fd355b54e4 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:13.011080 877 raft_consensus.cc:348] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:13.011261 876 ts_tablet_manager.cc:1398] T b8b7b807126c40939ef272fd355b54e4 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:18:13.012373 884 tablet_bootstrap.cc:654] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.014576 877 raft_consensus.cc:374] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.015053 877 raft_consensus.cc:729] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.015604 877 consensus_queue.cc:260] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:13.017110 885 tablet_bootstrap.cc:492] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:13.017477 885 ts_tablet_manager.cc:1367] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.005s
I20240502 08:18:13.018162 885 raft_consensus.cc:348] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:13.018410 885 raft_consensus.cc:374] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.018535 885 raft_consensus.cc:729] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.018878 885 consensus_queue.cc:260] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:13.020871 885 ts_tablet_manager.cc:1398] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:13.021157 878 tablet_bootstrap.cc:492] T b8b7b807126c40939ef272fd355b54e4 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:13.021351 878 ts_tablet_manager.cc:1367] T b8b7b807126c40939ef272fd355b54e4 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.074s user 0.001s sys 0.010s
I20240502 08:18:13.022168 878 raft_consensus.cc:348] T b8b7b807126c40939ef272fd355b54e4 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:13.022414 878 raft_consensus.cc:374] T b8b7b807126c40939ef272fd355b54e4 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.022548 878 raft_consensus.cc:729] T b8b7b807126c40939ef272fd355b54e4 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.022816 878 consensus_queue.cc:260] T b8b7b807126c40939ef272fd355b54e4 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:13.023085 884 tablet_bootstrap.cc:492] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:13.023257 884 ts_tablet_manager.cc:1367] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240502 08:18:13.024024 884 raft_consensus.cc:348] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:13.024277 884 raft_consensus.cc:374] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.024374 884 raft_consensus.cc:729] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.024582 884 consensus_queue.cc:260] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:13.026769 878 ts_tablet_manager.cc:1398] T b8b7b807126c40939ef272fd355b54e4 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:13.028131 878 tablet_bootstrap.cc:492] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:13.030025 877 ts_tablet_manager.cc:1398] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240502 08:18:13.030545 884 ts_tablet_manager.cc:1398] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 08:18:13.032306 878 tablet_bootstrap.cc:654] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.045753 878 tablet_bootstrap.cc:492] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:13.046028 878 ts_tablet_manager.cc:1367] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20240502 08:18:13.047169 878 raft_consensus.cc:348] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:13.047375 878 raft_consensus.cc:374] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.047492 878 raft_consensus.cc:729] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.047729 878 consensus_queue.cc:260] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:13.048319 878 ts_tablet_manager.cc:1398] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:13.066908 30159 consensus_queue.cc:237] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:13.069571 30028 raft_consensus.cc:1261] T a40b618e4cb9426c902c3384dea3bb46 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.070223 861 consensus_queue.cc:1035] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.071097 30421 raft_consensus.cc:1261] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:13.072122 30096 consensus_peers.cc:485] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 -> Peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Couldn't send request to peer bf8ac71b15f740a0a928911360773442. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a40b618e4cb9426c902c3384dea3bb46. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:13.073097 32172 raft_consensus.cc:482] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:13.073217 32172 raft_consensus.cc:504] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:13.073801 32172 leader_election.cc:288] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:13.074697 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a896dba4b8b34b6da87cf9dd401dbf5d" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:13.074766 861 consensus_queue.cc:1035] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.074988 30289 raft_consensus.cc:2454] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:18:13.075685 27191 leader_election.cc:302] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:18:13.075703 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a896dba4b8b34b6da87cf9dd401dbf5d" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:13.075927 30421 raft_consensus.cc:2454] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:18:13.079259 32172 raft_consensus.cc:2781] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:13.079351 32172 raft_consensus.cc:482] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:13.079420 32172 raft_consensus.cc:3037] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.081349 32172 raft_consensus.cc:504] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:13.081818 32172 leader_election.cc:288] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:13.082232 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a896dba4b8b34b6da87cf9dd401dbf5d" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:13.082273 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a896dba4b8b34b6da87cf9dd401dbf5d" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:13.082453 30423 raft_consensus.cc:3037] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.082456 30289 raft_consensus.cc:3037] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.085147 30289 raft_consensus.cc:2454] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:18:13.085215 30423 raft_consensus.cc:2454] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:18:13.086139 27191 leader_election.cc:302] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters:
I20240502 08:18:13.086971 32172 raft_consensus.cc:2781] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:13.087291 32172 raft_consensus.cc:686] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:18:13.087553 32172 consensus_queue.cc:237] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:13.090986 861 raft_consensus.cc:2932] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:13.093432 27113 catalog_manager.cc:5484] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:13.095664 30028 raft_consensus.cc:2932] T a40b618e4cb9426c902c3384dea3bb46 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:13.097723 27116 catalog_manager.cc:5484] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:13.113440 30422 raft_consensus.cc:2932] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:13.130980 30159 consensus_queue.cc:237] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:13.132967 30289 raft_consensus.cc:1261] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.133046 30028 raft_consensus.cc:1261] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.133549 300 consensus_queue.cc:1035] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.133823 861 consensus_queue.cc:1035] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:13.134132 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57505255575249e698bf35929619423d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:13.136270 861 raft_consensus.cc:2932] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:13.136889 30289 raft_consensus.cc:2932] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:13.138273 30028 raft_consensus.cc:2932] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:13.141403 27115 catalog_manager.cc:5484] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:13.161252 30159 consensus_queue.cc:237] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:13.163379 27648 raft_consensus.cc:1261] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.163401 30289 raft_consensus.cc:1261] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:13.163501 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16b897307b4c4cbea72f814f64f1e7d7. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:13.163946 863 consensus_queue.cc:1035] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.164245 861 consensus_queue.cc:1035] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.166576 861 raft_consensus.cc:2932] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:13.167392 30289 raft_consensus.cc:2932] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:13.167819 27648 raft_consensus.cc:2932] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:13.177049 27116 catalog_manager.cc:5484] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:13.182087 867 raft_consensus.cc:482] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:13.182256 867 raft_consensus.cc:504] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:13.183171 867 leader_election.cc:288] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:13.184370 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed6b8adc62f471696cbf0330524e353" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:13.184610 30159 raft_consensus.cc:2454] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:18:13.186465 27326 leader_election.cc:302] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:13.186717 867 raft_consensus.cc:2781] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:13.186780 867 raft_consensus.cc:482] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:13.186846 867 raft_consensus.cc:3037] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.189678 867 raft_consensus.cc:504] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:13.190248 867 leader_election.cc:288] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:13.190659 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed6b8adc62f471696cbf0330524e353" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:13.190827 30159 raft_consensus.cc:3037] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.191277 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed6b8adc62f471696cbf0330524e353" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:18:13.191519 27516 raft_consensus.cc:3037] T 0ed6b8adc62f471696cbf0330524e353 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.191805 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed6b8adc62f471696cbf0330524e353" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:18:13.192976 30159 raft_consensus.cc:2454] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:18:13.193305 27326 leader_election.cc:302] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:13.193621 867 raft_consensus.cc:2781] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:13.193863 867 raft_consensus.cc:686] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:18:13.194111 867 consensus_queue.cc:237] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:13.194492 27516 raft_consensus.cc:2454] T 0ed6b8adc62f471696cbf0330524e353 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:18:13.201419 27115 catalog_manager.cc:5484] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:13.202625 27113 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:13.206125 27516 consensus_queue.cc:237] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:13.208137 30289 raft_consensus.cc:1261] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.208297 30422 raft_consensus.cc:1261] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.209362 837 consensus_queue.cc:1035] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.209780 837 consensus_queue.cc:1035] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:13.212348 27455 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 -> Peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): Couldn't send request to peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de4b822c4aca4f7f92b73cc9279ddec5. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:13.213801 27115 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet a896dba4b8b34b6da87cf9dd401dbf5d (table rebalancer_dynamic_table [id=0b8cec7dd5e64eafaf141c89271018ab]). Expected version 1 got 0
I20240502 08:18:13.214008 30289 raft_consensus.cc:1261] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.214491 835 tablet.cc:1722] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:13.214520 650 consensus_queue.cc:1035] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.214907 835 tablet.cc:1620] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: MemRowSet was empty: no flush needed.
I20240502 08:18:13.219609 836 tablet.cc:1722] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:13.220067 836 tablet.cc:1620] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:13.222277 30159 raft_consensus.cc:1261] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.222870 650 consensus_queue.cc:1035] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.225819 27115 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet a896dba4b8b34b6da87cf9dd401dbf5d (table rebalancer_dynamic_table [id=0b8cec7dd5e64eafaf141c89271018ab]). Expected version 1 got 0
I20240502 08:18:13.228430 27113 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=0b8cec7dd5e64eafaf141c89271018ab] alter complete (version 1)
I20240502 08:18:13.232441 27116 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet a896dba4b8b34b6da87cf9dd401dbf5d (table rebalancer_dynamic_table [id=0b8cec7dd5e64eafaf141c89271018ab]). Expected version 1 got 0
I20240502 08:18:13.250315 931 tablet.cc:1722] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:13.251536 931 tablet.cc:1620] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:13.252769 27116 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet a896dba4b8b34b6da87cf9dd401dbf5d (table rebalancer_dynamic_table [id=0b8cec7dd5e64eafaf141c89271018ab]). Expected version 1 got 0
I20240502 08:18:13.255561 650 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:13.259279 30159 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:13.262849 27116 catalog_manager.cc:5484] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:13.273794 30288 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:13.296236 30422 consensus_queue.cc:237] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:13.298275 30159 raft_consensus.cc:1261] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.298369 27516 raft_consensus.cc:1261] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:13.298940 30358 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Couldn't send request to peer bf8ac71b15f740a0a928911360773442. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fefcc470084f4ec9be6ba45d3ba22ce5. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:13.299120 632 consensus_queue.cc:1035] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.299624 632 consensus_queue.cc:1035] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.314247 616 raft_consensus.cc:2932] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:13.316733 27516 raft_consensus.cc:2932] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:13.321427 27116 catalog_manager.cc:5484] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:13.321652 30159 raft_consensus.cc:2932] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:13.335587 300 raft_consensus.cc:482] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:13.335772 300 raft_consensus.cc:504] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:13.336401 300 leader_election.cc:288] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:13.336812 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4165b86bc05e47f6a9803fb45ca01569" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:18:13.337040 27516 raft_consensus.cc:2454] T 4165b86bc05e47f6a9803fb45ca01569 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:13.337237 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4165b86bc05e47f6a9803fb45ca01569" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:18:13.337426 27388 raft_consensus.cc:2454] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:13.337420 30096 leader_election.cc:302] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:13.337693 300 raft_consensus.cc:2781] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:13.337764 300 raft_consensus.cc:482] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:13.337832 300 raft_consensus.cc:3037] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.339738 300 raft_consensus.cc:504] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:13.340209 27113 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:13.342243 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4165b86bc05e47f6a9803fb45ca01569" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:18:13.342473 27516 raft_consensus.cc:3037] T 4165b86bc05e47f6a9803fb45ca01569 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.342823 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4165b86bc05e47f6a9803fb45ca01569" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:18:13.342990 27388 raft_consensus.cc:3037] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.343794 300 leader_election.cc:288] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:13.345373 27516 raft_consensus.cc:2454] T 4165b86bc05e47f6a9803fb45ca01569 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:13.345486 27388 raft_consensus.cc:2454] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:13.346220 30096 leader_election.cc:302] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:13.350019 300 raft_consensus.cc:2781] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:13.350219 300 raft_consensus.cc:686] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:13.350509 300 consensus_queue.cc:237] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:13.353842 835 tablet.cc:1722] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:13.354431 835 tablet.cc:1620] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: MemRowSet was empty: no flush needed.
I20240502 08:18:13.354521 931 tablet.cc:1722] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:13.354964 931 tablet.cc:1620] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:13.360451 836 tablet.cc:1722] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:13.360903 836 tablet.cc:1620] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:13.365626 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=0b8cec7dd5e64eafaf141c89271018ab] alter complete (version 2)
I20240502 08:18:13.369832 27257 consensus_queue.cc:237] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:13.373972 30028 raft_consensus.cc:1261] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.374904 30159 raft_consensus.cc:1261] T a9e5e67e3fe54491841b33d0d4bd047b P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.375450 32172 consensus_queue.cc:1035] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.376313 837 consensus_queue.cc:1035] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:13.385237 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9e5e67e3fe54491841b33d0d4bd047b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:13.386152 861 raft_consensus.cc:482] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:13.386320 861 raft_consensus.cc:504] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:13.386971 861 leader_election.cc:288] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:13.387737 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b7b807126c40939ef272fd355b54e4" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:13.387969 30027 raft_consensus.cc:2454] T b8b7b807126c40939ef272fd355b54e4 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:13.388446 30093 leader_election.cc:302] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:13.388995 916 raft_consensus.cc:2781] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:13.389088 916 raft_consensus.cc:482] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:13.389183 916 raft_consensus.cc:3037] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.392112 916 raft_consensus.cc:504] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:13.392725 916 leader_election.cc:288] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:13.395220 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b7b807126c40939ef272fd355b54e4" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:13.395431 30289 raft_consensus.cc:3037] T b8b7b807126c40939ef272fd355b54e4 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.395695 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b7b807126c40939ef272fd355b54e4" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:13.395874 30025 raft_consensus.cc:3037] T b8b7b807126c40939ef272fd355b54e4 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.398123 30289 raft_consensus.cc:2454] T b8b7b807126c40939ef272fd355b54e4 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:13.401693 27113 catalog_manager.cc:5484] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:13.404623 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b7b807126c40939ef272fd355b54e4" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:13.404901 30288 raft_consensus.cc:2362] T b8b7b807126c40939ef272fd355b54e4 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1. Re-sending same reply.
I20240502 08:18:13.405481 30093 leader_election.cc:302] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:13.405850 300 raft_consensus.cc:2781] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:13.406755 300 raft_consensus.cc:686] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:13.407152 300 consensus_queue.cc:237] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:13.407334 837 raft_consensus.cc:2932] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:13.412201 30025 raft_consensus.cc:2454] T b8b7b807126c40939ef272fd355b54e4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:13.413286 30157 raft_consensus.cc:2932] T a9e5e67e3fe54491841b33d0d4bd047b P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:13.420262 27113 catalog_manager.cc:5484] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:13.426234 27113 catalog_manager.cc:5484] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:13.428294 30025 raft_consensus.cc:2932] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:13.460368 30422 consensus_queue.cc:237] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:13.463307 27650 raft_consensus.cc:1261] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:13.463770 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Couldn't send request to peer 84cda20738d2452da166e31fc6da57a6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c78e068ec3a451c892d58174b3f837c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:13.463821 30288 raft_consensus.cc:1261] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.464067 390 consensus_queue.cc:1035] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.464342 632 consensus_queue.cc:1035] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.466284 390 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:13.466758 27650 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:13.468724 27113 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:13.471065 30288 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:13.472726 27116 catalog_manager.cc:5484] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index -1 to 2, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:13.482896 835 tablet.cc:1722] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:13.484017 931 tablet.cc:1722] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:13.491490 836 tablet.cc:1722] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:13.492165 27113 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=0b8cec7dd5e64eafaf141c89271018ab] alter complete (version 3)
I20240502 08:18:13.507273 30288 consensus_queue.cc:237] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:13.509807 967 ts_tablet_manager.cc:902] T a40b618e4cb9426c902c3384dea3bb46 P bf8ac71b15f740a0a928911360773442: Initiating tablet copy from peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:13.510162 30157 raft_consensus.cc:1261] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.510244 27516 raft_consensus.cc:1261] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.510829 966 consensus_queue.cc:1035] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.511240 626 consensus_queue.cc:1035] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:13.511716 30225 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 -> Peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): Couldn't send request to peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 757f65cd77bf49d882c2545673282e03. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:13.515120 27519 raft_consensus.cc:2932] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:13.513876 966 raft_consensus.cc:2932] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:13.533259 27115 catalog_manager.cc:5484] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:13.534296 30159 raft_consensus.cc:2932] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:13.547379 967 tablet_copy_client.cc:308] T a40b618e4cb9426c902c3384dea3bb46 P bf8ac71b15f740a0a928911360773442: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.6:38899
I20240502 08:18:13.548445 30180 tablet_copy_service.cc:140] P 84cda20738d2452da166e31fc6da57a6: Received BeginTabletCopySession request for tablet a40b618e4cb9426c902c3384dea3bb46 from peer bf8ac71b15f740a0a928911360773442 ({username='slave'} at 127.20.207.4:59449)
I20240502 08:18:13.548604 30180 tablet_copy_service.cc:161] P 84cda20738d2452da166e31fc6da57a6: Beginning new tablet copy session on tablet a40b618e4cb9426c902c3384dea3bb46 from peer bf8ac71b15f740a0a928911360773442 at {username='slave'} at 127.20.207.4:59449: session id = bf8ac71b15f740a0a928911360773442-a40b618e4cb9426c902c3384dea3bb46
I20240502 08:18:13.550413 974 ts_tablet_manager.cc:902] T 57505255575249e698bf35929619423d P 83e171a5e16c4008923f1d43fc5ce7b1: Initiating tablet copy from peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:13.551116 30180 tablet_copy_source_session.cc:205] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:13.552358 967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40b618e4cb9426c902c3384dea3bb46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.560350 967 tablet_copy_client.cc:783] T a40b618e4cb9426c902c3384dea3bb46 P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:13.560698 967 tablet_copy_client.cc:647] T a40b618e4cb9426c902c3384dea3bb46 P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:13.561187 974 tablet_copy_client.cc:308] T 57505255575249e698bf35929619423d P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.6:38899
I20240502 08:18:13.561859 30179 tablet_copy_service.cc:140] P 84cda20738d2452da166e31fc6da57a6: Received BeginTabletCopySession request for tablet 57505255575249e698bf35929619423d from peer 83e171a5e16c4008923f1d43fc5ce7b1 ({username='slave'} at 127.20.207.2:56535)
I20240502 08:18:13.562054 30179 tablet_copy_service.cc:161] P 84cda20738d2452da166e31fc6da57a6: Beginning new tablet copy session on tablet 57505255575249e698bf35929619423d from peer 83e171a5e16c4008923f1d43fc5ce7b1 at {username='slave'} at 127.20.207.2:56535: session id = 83e171a5e16c4008923f1d43fc5ce7b1-57505255575249e698bf35929619423d
I20240502 08:18:13.562245 967 tablet_copy_client.cc:523] T a40b618e4cb9426c902c3384dea3bb46 P bf8ac71b15f740a0a928911360773442: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:13.562281 30288 consensus_queue.cc:237] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:13.564788 30421 raft_consensus.cc:1261] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.565035 30179 tablet_copy_source_session.cc:205] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:13.565248 966 consensus_queue.cc:1035] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.565558 967 tablet_bootstrap.cc:492] T a40b618e4cb9426c902c3384dea3bb46 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:13.566243 27388 raft_consensus.cc:1261] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.566802 966 consensus_queue.cc:1035] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:13.567402 30226 consensus_peers.cc:485] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 -> Peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Couldn't send request to peer bf8ac71b15f740a0a928911360773442. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1008f7127704fe8b3eb6f9246b0a79c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:13.569160 974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57505255575249e698bf35929619423d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.570298 623 raft_consensus.cc:2932] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:13.578011 27115 catalog_manager.cc:5484] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:13.578272 27390 raft_consensus.cc:2932] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:13.580771 967 tablet_bootstrap.cc:492] T a40b618e4cb9426c902c3384dea3bb46 P bf8ac71b15f740a0a928911360773442: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:13.581164 967 tablet_bootstrap.cc:492] T a40b618e4cb9426c902c3384dea3bb46 P bf8ac71b15f740a0a928911360773442: Bootstrap complete.
I20240502 08:18:13.581403 967 ts_tablet_manager.cc:1367] T a40b618e4cb9426c902c3384dea3bb46 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20240502 08:18:13.582260 967 raft_consensus.cc:348] T a40b618e4cb9426c902c3384dea3bb46 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:13.582527 967 raft_consensus.cc:729] T a40b618e4cb9426c902c3384dea3bb46 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: LEARNER
I20240502 08:18:13.582720 967 consensus_queue.cc:260] T a40b618e4cb9426c902c3384dea3bb46 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:13.583235 967 ts_tablet_manager.cc:1398] T a40b618e4cb9426c902c3384dea3bb46 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:18:13.590090 30421 raft_consensus.cc:2932] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:13.591061 30179 tablet_copy_service.cc:342] P 84cda20738d2452da166e31fc6da57a6: Request end of tablet copy session bf8ac71b15f740a0a928911360773442-a40b618e4cb9426c902c3384dea3bb46 received from {username='slave'} at 127.20.207.4:59449
I20240502 08:18:13.591218 30179 tablet_copy_service.cc:433] P 84cda20738d2452da166e31fc6da57a6: ending tablet copy session bf8ac71b15f740a0a928911360773442-a40b618e4cb9426c902c3384dea3bb46 on tablet a40b618e4cb9426c902c3384dea3bb46 with peer bf8ac71b15f740a0a928911360773442
I20240502 08:18:13.595952 988 ts_tablet_manager.cc:902] T 16b897307b4c4cbea72f814f64f1e7d7 P b01e7c8a302d41078fa39cfef665ff59: Initiating tablet copy from peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:13.597369 27113 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:13.597498 27113 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:13.597486 974 tablet_copy_client.cc:783] T 57505255575249e698bf35929619423d P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:13.597693 974 tablet_copy_client.cc:647] T 57505255575249e698bf35929619423d P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:13.599756 974 tablet_copy_client.cc:523] T 57505255575249e698bf35929619423d P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:13.600764 27113 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet a896dba4b8b34b6da87cf9dd401dbf5d
I20240502 08:18:13.602051 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet a896dba4b8b34b6da87cf9dd401dbf5d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:13 UTC) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:13.602368 30400 tablet_service.cc:1514] Processing DeleteTablet for tablet a896dba4b8b34b6da87cf9dd401dbf5d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:13 UTC) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:13.602890 974 tablet_bootstrap.cc:492] T 57505255575249e698bf35929619423d P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:13.605965 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet a896dba4b8b34b6da87cf9dd401dbf5d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:13 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:18:13.613086 995 tablet_replica.cc:330] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:13.613415 996 tablet_replica.cc:330] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:18:13.613938 994 tablet_replica.cc:330] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:13.614239 994 raft_consensus.cc:2229] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:13.614161 771 consensus_queue.cc:1035] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.614567 994 raft_consensus.cc:2258] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:13.614992 996 raft_consensus.cc:2229] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:13.615253 995 raft_consensus.cc:2229] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:13.615568 996 raft_consensus.cc:2258] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:13.615809 994 ts_tablet_manager.cc:1875] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:13.617062 995 raft_consensus.cc:2258] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:13.618124 996 ts_tablet_manager.cc:1875] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:13.619869 995 ts_tablet_manager.cc:1875] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:13.625844 994 ts_tablet_manager.cc:1888] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:13.627151 996 ts_tablet_manager.cc:1888] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:13.631987 996 log.cc:1196] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/a896dba4b8b34b6da87cf9dd401dbf5d
I20240502 08:18:13.632213 867 consensus_queue.cc:1035] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.632741 996 ts_tablet_manager.cc:1909] T a896dba4b8b34b6da87cf9dd401dbf5d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:18:13.633937 994 log.cc:1196] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/a896dba4b8b34b6da87cf9dd401dbf5d
I20240502 08:18:13.635792 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet a896dba4b8b34b6da87cf9dd401dbf5d (table rebalancer_dynamic_table_new_name [id=0b8cec7dd5e64eafaf141c89271018ab]) successfully deleted
I20240502 08:18:13.638522 994 ts_tablet_manager.cc:1909] T a896dba4b8b34b6da87cf9dd401dbf5d P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting consensus metadata
I20240502 08:18:13.639835 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet a896dba4b8b34b6da87cf9dd401dbf5d (table rebalancer_dynamic_table_new_name [id=0b8cec7dd5e64eafaf141c89271018ab]) successfully deleted
I20240502 08:18:13.642310 995 ts_tablet_manager.cc:1888] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:13.642517 995 log.cc:1196] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/a896dba4b8b34b6da87cf9dd401dbf5d
I20240502 08:18:13.643283 995 ts_tablet_manager.cc:1909] T a896dba4b8b34b6da87cf9dd401dbf5d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting consensus metadata
I20240502 08:18:13.644588 988 tablet_copy_client.cc:308] T 16b897307b4c4cbea72f814f64f1e7d7 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.6:38899
I20240502 08:18:13.645251 974 tablet_bootstrap.cc:492] T 57505255575249e698bf35929619423d P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:13.645747 30179 tablet_copy_service.cc:140] P 84cda20738d2452da166e31fc6da57a6: Received BeginTabletCopySession request for tablet 16b897307b4c4cbea72f814f64f1e7d7 from peer b01e7c8a302d41078fa39cfef665ff59 ({username='slave'} at 127.20.207.3:35877)
I20240502 08:18:13.645918 30179 tablet_copy_service.cc:161] P 84cda20738d2452da166e31fc6da57a6: Beginning new tablet copy session on tablet 16b897307b4c4cbea72f814f64f1e7d7 from peer b01e7c8a302d41078fa39cfef665ff59 at {username='slave'} at 127.20.207.3:35877: session id = b01e7c8a302d41078fa39cfef665ff59-16b897307b4c4cbea72f814f64f1e7d7
I20240502 08:18:13.646785 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet a896dba4b8b34b6da87cf9dd401dbf5d (table rebalancer_dynamic_table_new_name [id=0b8cec7dd5e64eafaf141c89271018ab]) successfully deleted
I20240502 08:18:13.648885 30179 tablet_copy_source_session.cc:205] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:13.650307 988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b897307b4c4cbea72f814f64f1e7d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.657649 988 tablet_copy_client.cc:783] T 16b897307b4c4cbea72f814f64f1e7d7 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:13.657886 974 tablet_bootstrap.cc:492] T 57505255575249e698bf35929619423d P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap complete.
I20240502 08:18:13.658011 988 tablet_copy_client.cc:647] T 16b897307b4c4cbea72f814f64f1e7d7 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:13.658182 974 ts_tablet_manager.cc:1367] T 57505255575249e698bf35929619423d P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.002s
I20240502 08:18:13.658969 974 raft_consensus.cc:348] T 57505255575249e698bf35929619423d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:13.659252 974 raft_consensus.cc:729] T 57505255575249e698bf35929619423d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: LEARNER
I20240502 08:18:13.659498 974 consensus_queue.cc:260] T 57505255575249e698bf35929619423d P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:13.659945 988 tablet_copy_client.cc:523] T 16b897307b4c4cbea72f814f64f1e7d7 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:13.663359 974 ts_tablet_manager.cc:1398] T 57505255575249e698bf35929619423d P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 08:18:13.664873 988 tablet_bootstrap.cc:492] T 16b897307b4c4cbea72f814f64f1e7d7 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:13.666993 30179 tablet_copy_service.cc:342] P 84cda20738d2452da166e31fc6da57a6: Request end of tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-57505255575249e698bf35929619423d received from {username='slave'} at 127.20.207.2:56535
I20240502 08:18:13.667115 30179 tablet_copy_service.cc:433] P 84cda20738d2452da166e31fc6da57a6: ending tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-57505255575249e698bf35929619423d on tablet 57505255575249e698bf35929619423d with peer 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:13.675598 988 tablet_bootstrap.cc:492] T 16b897307b4c4cbea72f814f64f1e7d7 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:13.676570 988 tablet_bootstrap.cc:492] T 16b897307b4c4cbea72f814f64f1e7d7 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap complete.
I20240502 08:18:13.676900 988 ts_tablet_manager.cc:1367] T 16b897307b4c4cbea72f814f64f1e7d7 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240502 08:18:13.677508 988 raft_consensus.cc:348] T 16b897307b4c4cbea72f814f64f1e7d7 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:13.677878 988 raft_consensus.cc:729] T 16b897307b4c4cbea72f814f64f1e7d7 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: LEARNER
I20240502 08:18:13.678151 988 consensus_queue.cc:260] T 16b897307b4c4cbea72f814f64f1e7d7 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:13.680191 988 ts_tablet_manager.cc:1398] T 16b897307b4c4cbea72f814f64f1e7d7 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:13.680814 30179 tablet_copy_service.cc:342] P 84cda20738d2452da166e31fc6da57a6: Request end of tablet copy session b01e7c8a302d41078fa39cfef665ff59-16b897307b4c4cbea72f814f64f1e7d7 received from {username='slave'} at 127.20.207.3:35877
I20240502 08:18:13.680948 30179 tablet_copy_service.cc:433] P 84cda20738d2452da166e31fc6da57a6: ending tablet copy session b01e7c8a302d41078fa39cfef665ff59-16b897307b4c4cbea72f814f64f1e7d7 on tablet 16b897307b4c4cbea72f814f64f1e7d7 with peer b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:13.683542 30421 consensus_queue.cc:237] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:13.686136 27257 raft_consensus.cc:1261] T e5af418a0ae043a387b5030a00b47e91 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:13.686228 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5af418a0ae043a387b5030a00b47e91. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:13.686242 30025 raft_consensus.cc:1261] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.686693 616 consensus_queue.cc:1035] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.686942 390 consensus_queue.cc:1035] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.689604 632 raft_consensus.cc:2932] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:13.690733 30028 raft_consensus.cc:2932] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:13.691264 27257 raft_consensus.cc:2932] T e5af418a0ae043a387b5030a00b47e91 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:13.695747 27113 catalog_manager.cc:5484] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:13.707608 27115 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:13.714799 30400 tablet_service.cc:1467] Processing CreateTablet for tablet dd2298138c914526ba3825dc3a4ed56c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c733d5389c7342969dbf12e6bea513f8]), partition=
I20240502 08:18:13.715076 27238 tablet_service.cc:1467] Processing CreateTablet for tablet dd2298138c914526ba3825dc3a4ed56c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c733d5389c7342969dbf12e6bea513f8]), partition=
I20240502 08:18:13.715193 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd2298138c914526ba3825dc3a4ed56c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.715188 27500 tablet_service.cc:1467] Processing CreateTablet for tablet dd2298138c914526ba3825dc3a4ed56c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c733d5389c7342969dbf12e6bea513f8]), partition=
I20240502 08:18:13.715410 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd2298138c914526ba3825dc3a4ed56c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.715538 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd2298138c914526ba3825dc3a4ed56c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.722183 30025 consensus_queue.cc:237] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
W20240502 08:18:13.725742 29963 consensus_peers.cc:485] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): Couldn't send request to peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0dea1ac1af1b45cf9abfd5e5a1d910a1. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:13.727376 30288 raft_consensus.cc:1261] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.730161 819 consensus_queue.cc:1035] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.743013 30421 raft_consensus.cc:1261] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.744796 819 consensus_queue.cc:1035] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.760314 1020 tablet_bootstrap.cc:492] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:13.761610 1021 tablet_bootstrap.cc:492] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:13.761868 819 raft_consensus.cc:2932] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:13.763944 1021 tablet_bootstrap.cc:654] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.765161 1020 tablet_bootstrap.cc:654] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.767356 30288 raft_consensus.cc:2932] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:13.768198 1018 tablet_bootstrap.cc:492] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:13.768262 30421 raft_consensus.cc:2932] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:13.770239 1018 tablet_bootstrap.cc:654] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.775223 27115 catalog_manager.cc:5484] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:13.787375 1020 tablet_bootstrap.cc:492] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:13.787515 1020 ts_tablet_manager.cc:1367] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.003s
I20240502 08:18:13.788331 1020 raft_consensus.cc:348] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:13.788551 1020 raft_consensus.cc:374] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.788599 1020 raft_consensus.cc:729] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.788841 1020 consensus_queue.cc:260] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:13.789443 1020 ts_tablet_manager.cc:1398] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:13.790702 1021 tablet_bootstrap.cc:492] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:13.790877 1021 ts_tablet_manager.cc:1367] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.000s
I20240502 08:18:13.791536 1021 raft_consensus.cc:348] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:13.791807 1021 raft_consensus.cc:374] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.791919 1021 raft_consensus.cc:729] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.792142 1021 consensus_queue.cc:260] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:13.794013 1021 ts_tablet_manager.cc:1398] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:13.796736 974 ts_tablet_manager.cc:902] T a9e5e67e3fe54491841b33d0d4bd047b P 83e171a5e16c4008923f1d43fc5ce7b1: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:13.800052 1018 tablet_bootstrap.cc:492] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:13.800206 1018 ts_tablet_manager.cc:1367] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.005s
I20240502 08:18:13.802557 1032 ts_tablet_manager.cc:902] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:13.802904 300 consensus_queue.cc:1035] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.803814 1018 raft_consensus.cc:348] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:13.806092 1018 raft_consensus.cc:374] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.806242 1018 raft_consensus.cc:729] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.806533 1018 consensus_queue.cc:260] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:13.811216 1018 ts_tablet_manager.cc:1398] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240502 08:18:13.813582 967 ts_tablet_manager.cc:902] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442: Initiating tablet copy from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:13.825210 916 consensus_queue.cc:1035] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.850642 974 tablet_copy_client.cc:308] T a9e5e67e3fe54491841b33d0d4bd047b P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:18:13.851823 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet a9e5e67e3fe54491841b33d0d4bd047b from peer 83e171a5e16c4008923f1d43fc5ce7b1 ({username='slave'} at 127.20.207.2:44351)
I20240502 08:18:13.851962 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet a9e5e67e3fe54491841b33d0d4bd047b from peer 83e171a5e16c4008923f1d43fc5ce7b1 at {username='slave'} at 127.20.207.2:44351: session id = 83e171a5e16c4008923f1d43fc5ce7b1-a9e5e67e3fe54491841b33d0d4bd047b
I20240502 08:18:13.853713 967 tablet_copy_client.cc:308] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.8:41619
I20240502 08:18:13.854292 30443 tablet_copy_service.cc:140] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Received BeginTabletCopySession request for tablet fefcc470084f4ec9be6ba45d3ba22ce5 from peer bf8ac71b15f740a0a928911360773442 ({username='slave'} at 127.20.207.4:55985)
I20240502 08:18:13.854416 30443 tablet_copy_service.cc:161] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Beginning new tablet copy session on tablet fefcc470084f4ec9be6ba45d3ba22ce5 from peer bf8ac71b15f740a0a928911360773442 at {username='slave'} at 127.20.207.4:55985: session id = bf8ac71b15f740a0a928911360773442-fefcc470084f4ec9be6ba45d3ba22ce5
I20240502 08:18:13.854650 27278 tablet_copy_source_session.cc:205] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:13.855868 974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e5e67e3fe54491841b33d0d4bd047b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.857023 30443 tablet_copy_source_session.cc:205] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:13.858189 300 consensus_queue.cc:1035] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.858382 967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fefcc470084f4ec9be6ba45d3ba22ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.866530 974 tablet_copy_client.cc:783] T a9e5e67e3fe54491841b33d0d4bd047b P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:13.866801 974 tablet_copy_client.cc:647] T a9e5e67e3fe54491841b33d0d4bd047b P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:13.869699 974 tablet_copy_client.cc:523] T a9e5e67e3fe54491841b33d0d4bd047b P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:13.873521 974 tablet_bootstrap.cc:492] T a9e5e67e3fe54491841b33d0d4bd047b P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:13.874295 967 tablet_copy_client.cc:783] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:13.874528 967 tablet_copy_client.cc:647] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:13.875761 1032 tablet_copy_client.cc:308] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:13.876381 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet de4b822c4aca4f7f92b73cc9279ddec5 from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 ({username='slave'} at 127.20.207.1:38081)
I20240502 08:18:13.876492 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet de4b822c4aca4f7f92b73cc9279ddec5 from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 at {username='slave'} at 127.20.207.1:38081: session id = 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-de4b822c4aca4f7f92b73cc9279ddec5
I20240502 08:18:13.877511 967 tablet_copy_client.cc:523] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:13.879351 27541 tablet_copy_source_session.cc:205] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:13.880642 1032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de4b822c4aca4f7f92b73cc9279ddec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.882036 616 raft_consensus.cc:482] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:13.882217 967 tablet_bootstrap.cc:492] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:13.882180 616 raft_consensus.cc:504] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:13.882972 616 leader_election.cc:288] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:13.883493 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2298138c914526ba3825dc3a4ed56c" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:18:13.883687 27519 raft_consensus.cc:2454] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:13.883806 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2298138c914526ba3825dc3a4ed56c" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:13.884007 27258 raft_consensus.cc:2454] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:13.884482 30358 leader_election.cc:302] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:13.884754 390 raft_consensus.cc:2781] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:13.884904 390 raft_consensus.cc:482] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:13.885015 390 raft_consensus.cc:3037] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.888751 390 raft_consensus.cc:504] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:13.889707 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2298138c914526ba3825dc3a4ed56c" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:13.889881 27258 raft_consensus.cc:3037] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.892772 27258 raft_consensus.cc:2454] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:13.893301 30357 leader_election.cc:302] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters:
I20240502 08:18:13.893612 632 raft_consensus.cc:2781] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:13.893767 632 raft_consensus.cc:686] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:18:13.894045 632 consensus_queue.cc:237] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:13.894109 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2298138c914526ba3825dc3a4ed56c" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:18:13.894268 27519 raft_consensus.cc:3037] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.893764 390 leader_election.cc:288] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:13.896730 27519 raft_consensus.cc:2454] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:13.898357 1032 tablet_copy_client.cc:783] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:13.898697 1032 tablet_copy_client.cc:647] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:13.902936 27116 catalog_manager.cc:5484] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:13.904052 1032 tablet_copy_client.cc:523] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:13.913336 967 tablet_bootstrap.cc:492] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:13.916282 30421 consensus_queue.cc:237] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:13.919893 916 consensus_queue.cc:1035] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.927500 967 tablet_bootstrap.cc:492] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442: Bootstrap complete.
I20240502 08:18:13.927783 967 ts_tablet_manager.cc:1367] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.001s
I20240502 08:18:13.931881 27390 raft_consensus.cc:1203] T 57505255575249e698bf35929619423d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:13.936105 967 raft_consensus.cc:348] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:13.937145 967 raft_consensus.cc:729] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: LEARNER
I20240502 08:18:13.937364 967 consensus_queue.cc:260] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:13.942283 1032 tablet_bootstrap.cc:492] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:13.943253 967 ts_tablet_manager.cc:1398] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 08:18:13.943840 30443 tablet_copy_service.cc:342] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Request end of tablet copy session bf8ac71b15f740a0a928911360773442-fefcc470084f4ec9be6ba45d3ba22ce5 received from {username='slave'} at 127.20.207.4:55985
I20240502 08:18:13.943990 30443 tablet_copy_service.cc:433] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: ending tablet copy session bf8ac71b15f740a0a928911360773442-fefcc470084f4ec9be6ba45d3ba22ce5 on tablet fefcc470084f4ec9be6ba45d3ba22ce5 with peer bf8ac71b15f740a0a928911360773442
I20240502 08:18:13.944083 27258 raft_consensus.cc:1261] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.944931 632 consensus_queue.cc:1035] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:13.947172 30159 raft_consensus.cc:1261] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.949095 974 tablet_bootstrap.cc:492] T a9e5e67e3fe54491841b33d0d4bd047b P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:13.950318 974 tablet_bootstrap.cc:492] T a9e5e67e3fe54491841b33d0d4bd047b P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap complete.
I20240502 08:18:13.950670 974 ts_tablet_manager.cc:1367] T a9e5e67e3fe54491841b33d0d4bd047b P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.077s user 0.016s sys 0.019s
I20240502 08:18:13.951653 974 raft_consensus.cc:348] T a9e5e67e3fe54491841b33d0d4bd047b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:13.951952 974 raft_consensus.cc:729] T a9e5e67e3fe54491841b33d0d4bd047b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: LEARNER
I20240502 08:18:13.952159 974 consensus_queue.cc:260] T a9e5e67e3fe54491841b33d0d4bd047b P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:13.957511 632 consensus_queue.cc:1035] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:13.958822 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7064865693149b980ba54fb8e65bc48. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:13.962399 974 ts_tablet_manager.cc:1398] T a9e5e67e3fe54491841b33d0d4bd047b P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20240502 08:18:13.966450 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-a9e5e67e3fe54491841b33d0d4bd047b received from {username='slave'} at 127.20.207.2:44351
I20240502 08:18:13.966557 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-a9e5e67e3fe54491841b33d0d4bd047b on tablet a9e5e67e3fe54491841b33d0d4bd047b with peer 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:13.984225 27519 raft_consensus.cc:1203] T 16b897307b4c4cbea72f814f64f1e7d7 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:13.991796 27257 raft_consensus.cc:2932] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:13.989571 632 raft_consensus.cc:2932] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:13.996191 30159 raft_consensus.cc:2932] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:13.997285 1032 tablet_bootstrap.cc:492] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:13.997718 1032 tablet_bootstrap.cc:492] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap complete.
I20240502 08:18:13.998040 1032 ts_tablet_manager.cc:1367] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.056s user 0.010s sys 0.025s
I20240502 08:18:13.998808 1032 raft_consensus.cc:348] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:13.999106 1032 raft_consensus.cc:729] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: LEARNER
I20240502 08:18:13.999330 1032 consensus_queue.cc:260] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:14.008001 27115 catalog_manager.cc:5484] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:14.014861 1032 ts_tablet_manager.cc:1398] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20240502 08:18:14.015810 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-de4b822c4aca4f7f92b73cc9279ddec5 received from {username='slave'} at 127.20.207.1:38081
I20240502 08:18:14.015928 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-de4b822c4aca4f7f92b73cc9279ddec5 on tablet de4b822c4aca4f7f92b73cc9279ddec5 with peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60
I20240502 08:18:14.015944 967 ts_tablet_manager.cc:902] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442: Initiating tablet copy from peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:14.025825 27116 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:14.030222 27115 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:14.041653 30270 tablet_service.cc:1467] Processing CreateTablet for tablet cef42b07900c4b4e8589f284c011c300 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=50fcb176b4c44d52a482856f676a46d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.042232 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef42b07900c4b4e8589f284c011c300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.043502 30007 tablet_service.cc:1467] Processing CreateTablet for tablet cef42b07900c4b4e8589f284c011c300 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=50fcb176b4c44d52a482856f676a46d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.044003 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef42b07900c4b4e8589f284c011c300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.047030 30401 tablet_service.cc:1467] Processing CreateTablet for tablet 0524d946c6984ebcbf9bbd28d90c3844 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=50fcb176b4c44d52a482856f676a46d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.047408 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0524d946c6984ebcbf9bbd28d90c3844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.047816 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 0524d946c6984ebcbf9bbd28d90c3844 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=50fcb176b4c44d52a482856f676a46d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.048295 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0524d946c6984ebcbf9bbd28d90c3844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.050962 1032 ts_tablet_manager.cc:902] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Initiating tablet copy from peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:14.054168 1018 tablet_bootstrap.cc:492] T 0524d946c6984ebcbf9bbd28d90c3844 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:14.055814 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 2e9b191ce9234c028b990efd74a59e2e (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=50fcb176b4c44d52a482856f676a46d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.056110 1018 tablet_bootstrap.cc:654] T 0524d946c6984ebcbf9bbd28d90c3844 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.056216 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e9b191ce9234c028b990efd74a59e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.056473 27519 raft_consensus.cc:1261] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:14.056473 27258 raft_consensus.cc:1261] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:14.057420 27629 tablet_service.cc:1467] Processing CreateTablet for tablet cef42b07900c4b4e8589f284c011c300 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=50fcb176b4c44d52a482856f676a46d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.057627 616 consensus_queue.cc:1035] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:14.057763 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef42b07900c4b4e8589f284c011c300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.058362 616 consensus_queue.cc:1035] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:14.062767 30400 tablet_service.cc:1467] Processing CreateTablet for tablet 2e9b191ce9234c028b990efd74a59e2e (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=50fcb176b4c44d52a482856f676a46d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.063256 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e9b191ce9234c028b990efd74a59e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.065271 1020 tablet_bootstrap.cc:492] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:14.066543 30006 tablet_service.cc:1467] Processing CreateTablet for tablet 2e9b191ce9234c028b990efd74a59e2e (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=50fcb176b4c44d52a482856f676a46d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.066923 1020 tablet_bootstrap.cc:654] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.066964 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e9b191ce9234c028b990efd74a59e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.072088 30138 tablet_service.cc:1467] Processing CreateTablet for tablet 0524d946c6984ebcbf9bbd28d90c3844 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=50fcb176b4c44d52a482856f676a46d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.072454 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0524d946c6984ebcbf9bbd28d90c3844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.085467 27650 raft_consensus.cc:1203] T a40b618e4cb9426c902c3384dea3bb46 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:14.097332 1081 tablet.cc:1722] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:14.097906 1081 tablet.cc:1620] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59: MemRowSet was empty: no flush needed.
I20240502 08:18:14.100402 1018 tablet_bootstrap.cc:492] T 0524d946c6984ebcbf9bbd28d90c3844 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:14.100656 1018 ts_tablet_manager.cc:1367] T 0524d946c6984ebcbf9bbd28d90c3844 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.005s
I20240502 08:18:14.100714 27115 catalog_manager.cc:5576] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469) does not have the latest schema for tablet dd2298138c914526ba3825dc3a4ed56c (table rebalancer_dynamic_table [id=c733d5389c7342969dbf12e6bea513f8]). Expected version 1 got 0
I20240502 08:18:14.101585 1018 raft_consensus.cc:348] T 0524d946c6984ebcbf9bbd28d90c3844 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:14.101913 1018 raft_consensus.cc:374] T 0524d946c6984ebcbf9bbd28d90c3844 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.102080 1018 raft_consensus.cc:729] T 0524d946c6984ebcbf9bbd28d90c3844 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.102392 1018 consensus_queue.cc:260] T 0524d946c6984ebcbf9bbd28d90c3844 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:14.102991 1018 ts_tablet_manager.cc:1398] T 0524d946c6984ebcbf9bbd28d90c3844 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:14.134423 1085 tablet_bootstrap.cc:492] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:14.136798 1085 tablet_bootstrap.cc:654] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.149868 974 ts_tablet_manager.cc:902] T e5af418a0ae043a387b5030a00b47e91 P 83e171a5e16c4008923f1d43fc5ce7b1: Initiating tablet copy from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:14.149855 30288 consensus_queue.cc:237] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:14.150996 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=c733d5389c7342969dbf12e6bea513f8] alter complete (version 1)
I20240502 08:18:14.160938 1072 ts_tablet_manager.cc:902] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6: Initiating tablet copy from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
W20240502 08:18:14.163117 30226 consensus_peers.cc:485] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 -> Peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Couldn't send request to peer bf8ac71b15f740a0a928911360773442. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1f416f090284b8d92d53662cc7a3b97. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:14.164543 27390 raft_consensus.cc:1261] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:14.164773 1085 tablet_bootstrap.cc:492] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:14.164899 1085 ts_tablet_manager.cc:1367] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.000s
I20240502 08:18:14.165295 966 consensus_queue.cc:1035] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:14.165591 1085 raft_consensus.cc:348] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:14.165828 1085 raft_consensus.cc:374] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.165911 1085 raft_consensus.cc:729] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.166052 1087 tablet_bootstrap.cc:492] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:14.166216 1085 consensus_queue.cc:260] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:14.167665 1085 ts_tablet_manager.cc:1398] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:14.168306 1087 tablet_bootstrap.cc:654] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.187698 30419 raft_consensus.cc:1261] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:14.188302 966 consensus_queue.cc:1035] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:14.188402 1072 tablet_copy_client.cc:308] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.8:41619
I20240502 08:18:14.189325 30443 tablet_copy_service.cc:140] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Received BeginTabletCopySession request for tablet 4c78e068ec3a451c892d58174b3f837c from peer 84cda20738d2452da166e31fc6da57a6 ({username='slave'} at 127.20.207.6:44997)
I20240502 08:18:14.189491 30443 tablet_copy_service.cc:161] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Beginning new tablet copy session on tablet 4c78e068ec3a451c892d58174b3f837c from peer 84cda20738d2452da166e31fc6da57a6 at {username='slave'} at 127.20.207.6:44997: session id = 84cda20738d2452da166e31fc6da57a6-4c78e068ec3a451c892d58174b3f837c
I20240502 08:18:14.193580 30443 tablet_copy_source_session.cc:205] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:14.199401 1018 tablet_bootstrap.cc:492] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:14.200871 1072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c78e068ec3a451c892d58174b3f837c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.204897 27388 raft_consensus.cc:1203] T a9e5e67e3fe54491841b33d0d4bd047b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:14.208683 966 raft_consensus.cc:2932] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:14.213359 30419 raft_consensus.cc:2932] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:14.220546 27113 catalog_manager.cc:5484] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:14.256546 27116 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:14.294612 1018 tablet_bootstrap.cc:654] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.303957 916 raft_consensus.cc:482] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:14.304150 916 raft_consensus.cc:504] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:14.304852 916 leader_election.cc:288] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:14.305725 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0524d946c6984ebcbf9bbd28d90c3844" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:14.305724 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0524d946c6984ebcbf9bbd28d90c3844" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:14.305932 30422 raft_consensus.cc:2454] T 0524d946c6984ebcbf9bbd28d90c3844 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
W20240502 08:18:14.306313 30093 leader_election.cc:341] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:14.306766 30093 leader_election.cc:302] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:14.307385 916 raft_consensus.cc:2781] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:14.307492 916 raft_consensus.cc:482] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:14.307617 916 raft_consensus.cc:3037] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.311722 916 raft_consensus.cc:504] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:14.313047 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0524d946c6984ebcbf9bbd28d90c3844" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
W20240502 08:18:14.313530 30093 leader_election.cc:341] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:14.313592 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0524d946c6984ebcbf9bbd28d90c3844" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:14.313832 30422 raft_consensus.cc:3037] T 0524d946c6984ebcbf9bbd28d90c3844 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.319347 1104 ts_tablet_manager.cc:902] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Initiating tablet copy from peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:14.319849 30422 raft_consensus.cc:2454] T 0524d946c6984ebcbf9bbd28d90c3844 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:14.320314 30093 leader_election.cc:302] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:14.320653 861 raft_consensus.cc:2781] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:14.320785 861 raft_consensus.cc:686] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:14.321094 861 consensus_queue.cc:237] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:14.329974 916 leader_election.cc:288] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:14.329967 27116 catalog_manager.cc:5484] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:14.335435 1080 tablet.cc:1722] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:14.336243 1080 tablet.cc:1620] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:14.342382 1083 tablet_bootstrap.cc:492] T cef42b07900c4b4e8589f284c011c300 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:14.346480 1083 tablet_bootstrap.cc:654] T cef42b07900c4b4e8589f284c011c300 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.347450 1089 tablet_bootstrap.cc:492] T 0524d946c6984ebcbf9bbd28d90c3844 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:14.350165 1089 tablet_bootstrap.cc:654] T 0524d946c6984ebcbf9bbd28d90c3844 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
W20240502 08:18:14.352090 27584 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.5:37815, user_credentials={real_user=slave}} blocked reactor thread for 236575us
I20240502 08:18:14.352933 27116 catalog_manager.cc:5576] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505) does not have the latest schema for tablet dd2298138c914526ba3825dc3a4ed56c (table rebalancer_dynamic_table [id=c733d5389c7342969dbf12e6bea513f8]). Expected version 2 got 0
I20240502 08:18:14.373564 27388 raft_consensus.cc:2932] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:14.380028 27116 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet dd2298138c914526ba3825dc3a4ed56c (table rebalancer_dynamic_table [id=c733d5389c7342969dbf12e6bea513f8]). Expected version 2 got 1
I20240502 08:18:14.380630 1072 tablet_copy_client.cc:783] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:14.380903 1072 tablet_copy_client.cc:647] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:14.384052 1020 tablet_bootstrap.cc:492] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:14.384191 1020 ts_tablet_manager.cc:1367] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.319s user 0.005s sys 0.006s
I20240502 08:18:14.384943 1020 raft_consensus.cc:348] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:14.385221 1020 raft_consensus.cc:374] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.385269 1020 raft_consensus.cc:729] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.385478 1020 consensus_queue.cc:260] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:14.386433 1020 ts_tablet_manager.cc:1398] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:14.388326 27258 raft_consensus.cc:1203] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:14.401909 1072 tablet_copy_client.cc:523] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:14.425851 916 raft_consensus.cc:1050] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6: attempting to promote NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 to VOTER
I20240502 08:18:14.426824 916 consensus_queue.cc:237] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:14.427608 1072 tablet_bootstrap.cc:492] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:14.430029 27650 raft_consensus.cc:1203] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:14.430117 1087 tablet_bootstrap.cc:492] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:14.430275 1087 ts_tablet_manager.cc:1367] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.269s user 0.008s sys 0.026s
I20240502 08:18:14.431068 1087 raft_consensus.cc:348] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:14.431274 1087 raft_consensus.cc:374] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.431398 1087 raft_consensus.cc:729] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.431900 1087 consensus_queue.cc:260] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:14.439152 1089 tablet_bootstrap.cc:492] T 0524d946c6984ebcbf9bbd28d90c3844 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:14.441651 27390 raft_consensus.cc:1261] T 57505255575249e698bf35929619423d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:14.442520 30025 raft_consensus.cc:1261] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:14.440438 1032 tablet_copy_client.cc:308] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.7:40343
I20240502 08:18:14.443879 300 consensus_queue.cc:1035] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:14.444193 916 consensus_queue.cc:1035] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:14.442365 30288 raft_consensus.cc:1261] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:14.443442 30310 tablet_copy_service.cc:140] P 49c5c8f70e494edd9fcda40cc0af6cc4: Received BeginTabletCopySession request for tablet 757f65cd77bf49d882c2545673282e03 from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 ({username='slave'} at 127.20.207.1:49483)
I20240502 08:18:14.447396 30310 tablet_copy_service.cc:161] P 49c5c8f70e494edd9fcda40cc0af6cc4: Beginning new tablet copy session on tablet 757f65cd77bf49d882c2545673282e03 from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 at {username='slave'} at 127.20.207.1:49483: session id = 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-757f65cd77bf49d882c2545673282e03
I20240502 08:18:14.452316 27116 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet dd2298138c914526ba3825dc3a4ed56c (table rebalancer_dynamic_table [id=c733d5389c7342969dbf12e6bea513f8]). Expected version 2 got 1
I20240502 08:18:14.456125 30310 tablet_copy_source_session.cc:205] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:14.458892 300 consensus_queue.cc:1035] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:14.459200 1087 ts_tablet_manager.cc:1398] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.029s user 0.003s sys 0.000s
I20240502 08:18:14.461037 1087 tablet_bootstrap.cc:492] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:14.465224 974 tablet_copy_client.cc:308] T e5af418a0ae043a387b5030a00b47e91 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.8:41619
I20240502 08:18:14.465981 30443 tablet_copy_service.cc:140] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Received BeginTabletCopySession request for tablet e5af418a0ae043a387b5030a00b47e91 from peer 83e171a5e16c4008923f1d43fc5ce7b1 ({username='slave'} at 127.20.207.2:33397)
I20240502 08:18:14.466234 30443 tablet_copy_service.cc:161] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Beginning new tablet copy session on tablet e5af418a0ae043a387b5030a00b47e91 from peer 83e171a5e16c4008923f1d43fc5ce7b1 at {username='slave'} at 127.20.207.2:33397: session id = 83e171a5e16c4008923f1d43fc5ce7b1-e5af418a0ae043a387b5030a00b47e91
I20240502 08:18:14.468853 1018 tablet_bootstrap.cc:492] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:14.469887 1018 ts_tablet_manager.cc:1367] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.271s user 0.003s sys 0.001s
I20240502 08:18:14.469516 30443 tablet_copy_source_session.cc:205] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:14.471750 1018 raft_consensus.cc:348] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:14.473269 1018 raft_consensus.cc:374] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.473635 1018 raft_consensus.cc:729] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.474128 1018 consensus_queue.cc:260] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:14.480409 861 raft_consensus.cc:2932] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:14.483281 974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5af418a0ae043a387b5030a00b47e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.488269 27388 raft_consensus.cc:2932] T 57505255575249e698bf35929619423d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:14.488236 30288 raft_consensus.cc:2932] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:14.488972 1026 raft_consensus.cc:482] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:14.489112 1026 raft_consensus.cc:504] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:14.489800 1026 leader_election.cc:288] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:14.496665 27116 catalog_manager.cc:5484] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:14.501123 1087 tablet_bootstrap.cc:654] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.507822 30157 consensus_queue.cc:237] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:14.508270 1104 tablet_copy_client.cc:308] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.5:37815
I20240502 08:18:14.508522 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9b191ce9234c028b990efd74a59e2e" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:14.508734 27258 raft_consensus.cc:2454] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:14.509538 30048 tablet_copy_service.cc:140] P 1f0bf4bc3c5146539146b825b048bff3: Received BeginTabletCopySession request for tablet 0dea1ac1af1b45cf9abfd5e5a1d910a1 from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 ({username='slave'} at 127.20.207.1:49229)
I20240502 08:18:14.509601 29963 leader_election.cc:302] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 1f0bf4bc3c5146539146b825b048bff3; no voters:
I20240502 08:18:14.509699 30048 tablet_copy_service.cc:161] P 1f0bf4bc3c5146539146b825b048bff3: Beginning new tablet copy session on tablet 0dea1ac1af1b45cf9abfd5e5a1d910a1 from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 at {username='slave'} at 127.20.207.1:49229: session id = 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-0dea1ac1af1b45cf9abfd5e5a1d910a1
I20240502 08:18:14.510375 27388 raft_consensus.cc:1261] T 57505255575249e698bf35929619423d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:14.510875 916 consensus_queue.cc:1035] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:14.511270 1124 ts_tablet_manager.cc:902] T e7064865693149b980ba54fb8e65bc48 P b01e7c8a302d41078fa39cfef665ff59: Initiating tablet copy from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:14.512291 1026 raft_consensus.cc:2781] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:14.512403 1026 raft_consensus.cc:482] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:14.512419 30288 raft_consensus.cc:1261] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:14.512493 1026 raft_consensus.cc:3037] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.512631 30048 tablet_copy_source_session.cc:205] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:14.512915 861 consensus_queue.cc:1035] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:14.513823 300 raft_consensus.cc:2932] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:14.514258 30290 raft_consensus.cc:2932] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:14.517334 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 57505255575249e698bf35929619423d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:14.517673 1026 raft_consensus.cc:504] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:14.518332 1026 leader_election.cc:288] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:14.519975 1072 tablet_bootstrap.cc:492] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:14.519768 27388 raft_consensus.cc:2932] T 57505255575249e698bf35929619423d P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:14.521206 27115 catalog_manager.cc:5484] T 57505255575249e698bf35929619423d P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:14.523803 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9b191ce9234c028b990efd74a59e2e" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:14.524016 27258 raft_consensus.cc:3037] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.525537 30006 tablet_service.cc:1514] Processing DeleteTablet for tablet 57505255575249e698bf35929619423d with delete_type TABLET_DATA_TOMBSTONED (TS 1f0bf4bc3c5146539146b825b048bff3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:14.527289 1072 tablet_bootstrap.cc:492] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6: Bootstrap complete.
I20240502 08:18:14.527554 1072 ts_tablet_manager.cc:1367] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.100s user 0.002s sys 0.006s
I20240502 08:18:14.528510 1072 raft_consensus.cc:348] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:14.528815 1072 raft_consensus.cc:729] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: LEARNER
I20240502 08:18:14.529945 27258 raft_consensus.cc:2454] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:14.530455 1087 tablet_bootstrap.cc:492] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:14.530375 29963 leader_election.cc:302] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 1f0bf4bc3c5146539146b825b048bff3; no voters:
I20240502 08:18:14.530589 861 raft_consensus.cc:1050] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6: attempting to promote NON_VOTER bf8ac71b15f740a0a928911360773442 to VOTER
I20240502 08:18:14.530644 1087 ts_tablet_manager.cc:1367] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.003s
I20240502 08:18:14.530859 819 raft_consensus.cc:2781] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:14.531002 819 raft_consensus.cc:686] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:18:14.531256 819 consensus_queue.cc:237] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:14.531450 1087 raft_consensus.cc:348] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:14.531378 861 consensus_queue.cc:237] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:14.531630 1087 raft_consensus.cc:374] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.531702 1087 raft_consensus.cc:729] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.531949 1087 consensus_queue.cc:260] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:14.533694 1087 ts_tablet_manager.cc:1398] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:14.529090 1072 consensus_queue.cc:260] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:14.539597 1072 ts_tablet_manager.cc:1398] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.012s user 0.000s sys 0.003s
I20240502 08:18:14.540540 1089 ts_tablet_manager.cc:1367] T 0524d946c6984ebcbf9bbd28d90c3844 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.193s user 0.002s sys 0.002s
I20240502 08:18:14.541726 1089 raft_consensus.cc:348] T 0524d946c6984ebcbf9bbd28d90c3844 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:14.542044 1089 raft_consensus.cc:374] T 0524d946c6984ebcbf9bbd28d90c3844 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.542181 1089 raft_consensus.cc:729] T 0524d946c6984ebcbf9bbd28d90c3844 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.542481 1089 consensus_queue.cc:260] T 0524d946c6984ebcbf9bbd28d90c3844 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:14.543586 632 raft_consensus.cc:482] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:14.543789 632 raft_consensus.cc:504] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:14.545070 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9b191ce9234c028b990efd74a59e2e" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:14.545940 632 leader_election.cc:288] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:14.546578 1018 ts_tablet_manager.cc:1398] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.076s user 0.002s sys 0.000s
I20240502 08:18:14.546614 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9b191ce9234c028b990efd74a59e2e" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:14.546914 27255 raft_consensus.cc:2379] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in current term 1: Already voted for candidate 1f0bf4bc3c5146539146b825b048bff3 in this term.
I20240502 08:18:14.546702 27258 consensus_queue.cc:237] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:14.551688 1104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dea1ac1af1b45cf9abfd5e5a1d910a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.547406 30357 leader_election.cc:302] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:18:14.554466 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9b191ce9234c028b990efd74a59e2e" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:14.554724 30422 raft_consensus.cc:2454] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:14.540448 916 raft_consensus.cc:1050] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6: attempting to promote NON_VOTER b01e7c8a302d41078fa39cfef665ff59 to VOTER
I20240502 08:18:14.555267 30419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9b191ce9234c028b990efd74a59e2e" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:14.555477 30419 raft_consensus.cc:3037] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.556532 30443 tablet_copy_service.cc:342] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Request end of tablet copy session 84cda20738d2452da166e31fc6da57a6-4c78e068ec3a451c892d58174b3f837c received from {username='slave'} at 127.20.207.6:44997
I20240502 08:18:14.556679 30443 tablet_copy_service.cc:433] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: ending tablet copy session 84cda20738d2452da166e31fc6da57a6-4c78e068ec3a451c892d58174b3f837c on tablet 4c78e068ec3a451c892d58174b3f837c with peer 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:14.557457 1032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757f65cd77bf49d882c2545673282e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.559274 30419 raft_consensus.cc:2454] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:14.559608 632 raft_consensus.cc:2726] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:14.567891 916 consensus_queue.cc:237] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
W20240502 08:18:14.571033 967 thread.cc:636] tablet-download-pool-b1008f7127704fe8b3eb6f9246b0a79c [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.026s sys 0.082s
W20240502 08:18:14.572696 967 thread.cc:602] tablet-download-pool-b1008f7127704fe8b3eb6f9246b0a79c [worker] (thread pool) Time spent starting thread: real 0.510s user 0.026s sys 0.082s
I20240502 08:18:14.582659 1111 tablet.cc:1722] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:14.583495 1081 tablet.cc:1722] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:14.583974 1081 tablet.cc:1620] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59: MemRowSet was empty: no flush needed.
I20240502 08:18:14.587294 27116 catalog_manager.cc:5576] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505) does not have the latest schema for tablet dd2298138c914526ba3825dc3a4ed56c (table rebalancer_dynamic_table [id=c733d5389c7342969dbf12e6bea513f8]). Expected version 2 got 0
I20240502 08:18:14.594806 1080 tablet.cc:1722] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:14.595916 1080 tablet.cc:1620] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:14.596659 27109 catalog_manager.cc:5484] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:14.602208 27115 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=c733d5389c7342969dbf12e6bea513f8] alter complete (version 2)
I20240502 08:18:14.605515 1089 ts_tablet_manager.cc:1398] T 0524d946c6984ebcbf9bbd28d90c3844 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.065s user 0.000s sys 0.002s
I20240502 08:18:14.608275 1104 tablet_copy_client.cc:783] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:14.608523 1104 tablet_copy_client.cc:647] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:14.608740 967 tablet_copy_client.cc:308] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.7:40343
I20240502 08:18:14.611414 1032 tablet_copy_client.cc:783] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:14.612735 1104 tablet_copy_client.cc:523] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:14.613483 1032 tablet_copy_client.cc:647] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:14.615602 1133 tablet_replica.cc:330] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:14.616587 1133 raft_consensus.cc:2229] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:14.616987 1133 pending_rounds.cc:62] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3: Trying to abort 1 pending ops.
I20240502 08:18:14.612407 30310 tablet_copy_service.cc:140] P 49c5c8f70e494edd9fcda40cc0af6cc4: Received BeginTabletCopySession request for tablet b1008f7127704fe8b3eb6f9246b0a79c from peer bf8ac71b15f740a0a928911360773442 ({username='slave'} at 127.20.207.4:32967)
I20240502 08:18:14.617843 30310 tablet_copy_service.cc:161] P 49c5c8f70e494edd9fcda40cc0af6cc4: Beginning new tablet copy session on tablet b1008f7127704fe8b3eb6f9246b0a79c from peer bf8ac71b15f740a0a928911360773442 at {username='slave'} at 127.20.207.4:32967: session id = bf8ac71b15f740a0a928911360773442-b1008f7127704fe8b3eb6f9246b0a79c
I20240502 08:18:14.619426 1089 tablet_bootstrap.cc:492] T cef42b07900c4b4e8589f284c011c300 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:14.621073 30310 tablet_copy_source_session.cc:205] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:14.625886 1032 tablet_copy_client.cc:523] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:14.627403 1111 tablet.cc:1620] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: MemRowSet was empty: no flush needed.
I20240502 08:18:14.629484 1083 tablet_bootstrap.cc:492] T cef42b07900c4b4e8589f284c011c300 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:14.629742 1083 ts_tablet_manager.cc:1367] T cef42b07900c4b4e8589f284c011c300 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.287s user 0.006s sys 0.006s
I20240502 08:18:14.630564 967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1008f7127704fe8b3eb6f9246b0a79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.634575 27516 raft_consensus.cc:1261] T 16b897307b4c4cbea72f814f64f1e7d7 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240502 08:18:14.634688 30095 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.2:46291, user_credentials={real_user=slave}} blocked reactor thread for 81602.1us
I20240502 08:18:14.633888 1083 raft_consensus.cc:348] T cef42b07900c4b4e8589f284c011c300 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:14.635211 1132 consensus_queue.cc:1035] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:14.635331 1083 raft_consensus.cc:374] T cef42b07900c4b4e8589f284c011c300 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.635810 1083 raft_consensus.cc:729] T cef42b07900c4b4e8589f284c011c300 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.637087 974 tablet_copy_client.cc:783] T e5af418a0ae043a387b5030a00b47e91 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:14.637416 974 tablet_copy_client.cc:647] T e5af418a0ae043a387b5030a00b47e91 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:14.637733 27650 raft_consensus.cc:1261] T a40b618e4cb9426c902c3384dea3bb46 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:14.638273 1132 consensus_queue.cc:1035] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:14.617221 1133 pending_rounds.cc:68] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023156815569756160 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "57505255575249e698bf35929619423d" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } } }
I20240502 08:18:14.639477 1133 raft_consensus.cc:2894] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240502 08:18:14.639601 1133 raft_consensus.cc:2866] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 08:18:14.639709 1133 raft_consensus.cc:2258] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:14.641166 30288 raft_consensus.cc:1261] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:14.641722 32172 consensus_queue.cc:1035] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:14.642218 30290 raft_consensus.cc:1261] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:14.642287 27648 raft_consensus.cc:1261] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:14.644018 1132 consensus_queue.cc:1035] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 08:18:14.644083 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Couldn't send request to peer 1f0bf4bc3c5146539146b825b048bff3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2dfa509d47e54b0f9ccacc9954981fb6. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:14.644476 1132 consensus_queue.cc:1035] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:14.647866 32172 raft_consensus.cc:1050] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 to VOTER
I20240502 08:18:14.648713 32172 consensus_queue.cc:237] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:14.653290 30419 raft_consensus.cc:1261] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:14.653424 30025 raft_consensus.cc:1261] T a40b618e4cb9426c902c3384dea3bb46 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:14.654011 861 consensus_queue.cc:1035] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:14.654407 1132 consensus_queue.cc:1035] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:14.657454 1089 tablet_bootstrap.cc:654] T cef42b07900c4b4e8589f284c011c300 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.657615 1133 ts_tablet_manager.cc:1875] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:14.659482 27388 raft_consensus.cc:1261] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:14.660038 837 consensus_queue.cc:1035] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:14.662194 974 tablet_copy_client.cc:523] T e5af418a0ae043a387b5030a00b47e91 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:14.673904 974 tablet_bootstrap.cc:492] T e5af418a0ae043a387b5030a00b47e91 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:14.679580 1032 tablet_bootstrap.cc:492] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:14.686463 837 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:14.686986 27649 raft_consensus.cc:2932] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:14.690840 1138 raft_consensus.cc:2932] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:14.692453 30290 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:14.693420 27109 catalog_manager.cc:5484] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:14.695062 30159 raft_consensus.cc:1203] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:14.696727 27113 catalog_manager.cc:5484] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 6 to 7, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:14.703897 30290 raft_consensus.cc:3037] T 0524d946c6984ebcbf9bbd28d90c3844 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.707242 1151 ts_tablet_manager.cc:902] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442: Initiating tablet copy from peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:14.711061 861 consensus_queue.cc:1035] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:14.711403 1132 raft_consensus.cc:2932] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:14.717773 30418 raft_consensus.cc:2932] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:14.721576 30288 raft_consensus.cc:2932] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:14.724236 1032 tablet_bootstrap.cc:492] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:14.724591 1032 tablet_bootstrap.cc:492] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap complete.
I20240502 08:18:14.724809 1032 ts_tablet_manager.cc:1367] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.001s
I20240502 08:18:14.725647 1032 raft_consensus.cc:348] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:14.725895 1032 raft_consensus.cc:729] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: LEARNER
I20240502 08:18:14.720176 27109 catalog_manager.cc:5484] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:14.727277 27516 raft_consensus.cc:2932] T 16b897307b4c4cbea72f814f64f1e7d7 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:14.728221 27113 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:14.738452 30159 consensus_queue.cc:237] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:14.743134 27652 raft_consensus.cc:1261] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:14.743733 861 consensus_queue.cc:1035] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:14.746927 861 raft_consensus.cc:2932] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:14.750769 27388 raft_consensus.cc:1261] T a9e5e67e3fe54491841b33d0d4bd047b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:14.751286 837 consensus_queue.cc:1035] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:14.751292 27652 raft_consensus.cc:2932] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:14.757357 27114 catalog_manager.cc:5484] T 16b897307b4c4cbea72f814f64f1e7d7 P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:14.762198 1132 consensus_queue.cc:1035] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:14.762868 27389 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:14.769272 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 16b897307b4c4cbea72f814f64f1e7d7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:14.773476 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet 16b897307b4c4cbea72f814f64f1e7d7 with delete_type TABLET_DATA_TOMBSTONED (TS 49c5c8f70e494edd9fcda40cc0af6cc4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:14.788759 27519 raft_consensus.cc:1261] T 16b897307b4c4cbea72f814f64f1e7d7 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:14.790205 861 consensus_queue.cc:1035] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:14.792876 27519 raft_consensus.cc:2932] T 16b897307b4c4cbea72f814f64f1e7d7 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:14.796094 1032 consensus_queue.cc:260] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:14.806142 1032 ts_tablet_manager.cc:1398] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.081s user 0.004s sys 0.000s
I20240502 08:18:14.814342 30310 tablet_copy_service.cc:342] P 49c5c8f70e494edd9fcda40cc0af6cc4: Request end of tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-757f65cd77bf49d882c2545673282e03 received from {username='slave'} at 127.20.207.1:49483
I20240502 08:18:14.814564 30310 tablet_copy_service.cc:433] P 49c5c8f70e494edd9fcda40cc0af6cc4: ending tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-757f65cd77bf49d882c2545673282e03 on tablet 757f65cd77bf49d882c2545673282e03 with peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60
I20240502 08:18:14.673059 819 raft_consensus.cc:482] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:14.821386 30159 raft_consensus.cc:1261] T a9e5e67e3fe54491841b33d0d4bd047b P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:14.821434 819 raft_consensus.cc:504] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:14.822335 819 leader_election.cc:288] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:14.718343 30025 raft_consensus.cc:2932] T a40b618e4cb9426c902c3384dea3bb46 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:14.823194 930 consensus_queue.cc:1035] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:14.826081 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef42b07900c4b4e8589f284c011c300" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:14.826721 974 tablet_bootstrap.cc:492] T e5af418a0ae043a387b5030a00b47e91 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:14.827164 974 tablet_bootstrap.cc:492] T e5af418a0ae043a387b5030a00b47e91 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap complete.
I20240502 08:18:14.827466 974 ts_tablet_manager.cc:1367] T e5af418a0ae043a387b5030a00b47e91 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.154s user 0.011s sys 0.001s
I20240502 08:18:14.827677 30028 raft_consensus.cc:1261] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:14.828387 974 raft_consensus.cc:348] T e5af418a0ae043a387b5030a00b47e91 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:14.828708 974 raft_consensus.cc:729] T e5af418a0ae043a387b5030a00b47e91 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: LEARNER
I20240502 08:18:14.829062 974 consensus_queue.cc:260] T e5af418a0ae043a387b5030a00b47e91 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:14.831094 930 consensus_queue.cc:1035] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:14.832087 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef42b07900c4b4e8589f284c011c300" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
W20240502 08:18:14.832827 29961 leader_election.cc:341] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:14.833302 1151 tablet_copy_client.cc:308] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.7:40343
I20240502 08:18:14.834328 30310 tablet_copy_service.cc:140] P 49c5c8f70e494edd9fcda40cc0af6cc4: Received BeginTabletCopySession request for tablet f1f416f090284b8d92d53662cc7a3b97 from peer bf8ac71b15f740a0a928911360773442 ({username='slave'} at 127.20.207.4:32967)
I20240502 08:18:14.834662 30310 tablet_copy_service.cc:161] P 49c5c8f70e494edd9fcda40cc0af6cc4: Beginning new tablet copy session on tablet f1f416f090284b8d92d53662cc7a3b97 from peer bf8ac71b15f740a0a928911360773442 at {username='slave'} at 127.20.207.4:32967: session id = bf8ac71b15f740a0a928911360773442-f1f416f090284b8d92d53662cc7a3b97
I20240502 08:18:14.838529 30310 tablet_copy_source_session.cc:205] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:14.840004 1151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f416f090284b8d92d53662cc7a3b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.845309 32172 raft_consensus.cc:2932] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:14.849262 27650 raft_consensus.cc:2932] T a40b618e4cb9426c902c3384dea3bb46 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:14.850605 1165 tablet_replica.cc:330] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:14.850819 1165 raft_consensus.cc:2229] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:14.851075 1165 raft_consensus.cc:2258] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:14.851423 1111 tablet.cc:1722] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:14.851711 1151 tablet_copy_client.cc:783] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:14.851917 1165 ts_tablet_manager.cc:1875] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:14.851636 30025 raft_consensus.cc:2932] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:14.852175 1111 tablet.cc:1620] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: MemRowSet was empty: no flush needed.
I20240502 08:18:14.852706 1151 tablet_copy_client.cc:647] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:14.860131 27109 catalog_manager.cc:5576] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505) does not have the latest schema for tablet dd2298138c914526ba3825dc3a4ed56c (table rebalancer_dynamic_table_new_name [id=c733d5389c7342969dbf12e6bea513f8]). Expected version 3 got 1
I20240502 08:18:14.863998 27114 catalog_manager.cc:5576] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505) does not have the latest schema for tablet dd2298138c914526ba3825dc3a4ed56c (table rebalancer_dynamic_table_new_name [id=c733d5389c7342969dbf12e6bea513f8]). Expected version 3 got 2
I20240502 08:18:14.865254 1151 tablet_copy_client.cc:523] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:14.869199 1151 tablet_bootstrap.cc:492] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:14.871826 1081 tablet.cc:1722] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:14.874815 27114 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet dd2298138c914526ba3825dc3a4ed56c (table rebalancer_dynamic_table_new_name [id=c733d5389c7342969dbf12e6bea513f8]). Expected version 3 got 2
I20240502 08:18:14.876482 1111 tablet.cc:1722] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240502 08:18:14.882241 27326 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.3:44469, user_credentials={real_user=slave}} blocked reactor thread for 51915.6us
I20240502 08:18:14.885053 1151 tablet_bootstrap.cc:492] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:14.885560 1151 tablet_bootstrap.cc:492] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442: Bootstrap complete.
I20240502 08:18:14.885824 1151 ts_tablet_manager.cc:1367] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.007s
I20240502 08:18:14.886546 974 ts_tablet_manager.cc:1398] T e5af418a0ae043a387b5030a00b47e91 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.059s user 0.003s sys 0.000s
I20240502 08:18:14.886540 1151 raft_consensus.cc:348] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:14.886868 1151 raft_consensus.cc:729] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: LEARNER
I20240502 08:18:14.887194 30443 tablet_copy_service.cc:342] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Request end of tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-e5af418a0ae043a387b5030a00b47e91 received from {username='slave'} at 127.20.207.2:33397
I20240502 08:18:14.887316 30443 tablet_copy_service.cc:433] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: ending tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-e5af418a0ae043a387b5030a00b47e91 on tablet e5af418a0ae043a387b5030a00b47e91 with peer 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:14.887204 1151 consensus_queue.cc:260] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:14.887756 1151 ts_tablet_manager.cc:1398] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:14.889250 1104 tablet_bootstrap.cc:492] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:14.890139 27114 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=c733d5389c7342969dbf12e6bea513f8] alter complete (version 3)
I20240502 08:18:14.892127 1089 tablet_bootstrap.cc:492] T cef42b07900c4b4e8589f284c011c300 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:14.892304 1089 ts_tablet_manager.cc:1367] T cef42b07900c4b4e8589f284c011c300 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.273s user 0.010s sys 0.008s
I20240502 08:18:14.893086 1089 raft_consensus.cc:348] T cef42b07900c4b4e8589f284c011c300 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:14.893313 1089 raft_consensus.cc:374] T cef42b07900c4b4e8589f284c011c300 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.893393 1089 raft_consensus.cc:729] T cef42b07900c4b4e8589f284c011c300 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.893636 1089 consensus_queue.cc:260] T cef42b07900c4b4e8589f284c011c300 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:14.894409 1089 ts_tablet_manager.cc:1398] T cef42b07900c4b4e8589f284c011c300 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:14.900270 1083 consensus_queue.cc:260] T cef42b07900c4b4e8589f284c011c300 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:14.900576 30310 tablet_copy_service.cc:342] P 49c5c8f70e494edd9fcda40cc0af6cc4: Request end of tablet copy session bf8ac71b15f740a0a928911360773442-f1f416f090284b8d92d53662cc7a3b97 received from {username='slave'} at 127.20.207.4:32967
I20240502 08:18:14.900701 30310 tablet_copy_service.cc:433] P 49c5c8f70e494edd9fcda40cc0af6cc4: ending tablet copy session bf8ac71b15f740a0a928911360773442-f1f416f090284b8d92d53662cc7a3b97 on tablet f1f416f090284b8d92d53662cc7a3b97 with peer bf8ac71b15f740a0a928911360773442
I20240502 08:18:14.901278 27652 raft_consensus.cc:2454] T cef42b07900c4b4e8589f284c011c300 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:14.902082 29964 leader_election.cc:302] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:14.902199 1124 tablet_copy_client.cc:308] T e7064865693149b980ba54fb8e65bc48 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.8:41619
I20240502 08:18:14.902386 1129 raft_consensus.cc:2781] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:14.902483 1129 raft_consensus.cc:482] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:14.902575 1129 raft_consensus.cc:3037] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.902745 30443 tablet_copy_service.cc:140] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Received BeginTabletCopySession request for tablet e7064865693149b980ba54fb8e65bc48 from peer b01e7c8a302d41078fa39cfef665ff59 ({username='slave'} at 127.20.207.3:40171)
I20240502 08:18:14.902904 30443 tablet_copy_service.cc:161] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Beginning new tablet copy session on tablet e7064865693149b980ba54fb8e65bc48 from peer b01e7c8a302d41078fa39cfef665ff59 at {username='slave'} at 127.20.207.3:40171: session id = b01e7c8a302d41078fa39cfef665ff59-e7064865693149b980ba54fb8e65bc48
I20240502 08:18:14.902948 1083 ts_tablet_manager.cc:1398] T cef42b07900c4b4e8589f284c011c300 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.273s user 0.002s sys 0.000s
I20240502 08:18:14.903369 1080 tablet.cc:1722] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:14.905809 30443 tablet_copy_source_session.cc:205] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:14.906232 1129 raft_consensus.cc:504] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:14.907116 1124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7064865693149b980ba54fb8e65bc48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.909715 27109 catalog_manager.cc:5484] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:14.911129 1129 leader_election.cc:288] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:14.911105 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef42b07900c4b4e8589f284c011c300" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:14.911322 30288 raft_consensus.cc:3037] T cef42b07900c4b4e8589f284c011c300 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.911656 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef42b07900c4b4e8589f284c011c300" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:14.911849 27652 raft_consensus.cc:3037] T cef42b07900c4b4e8589f284c011c300 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.913888 30288 raft_consensus.cc:2454] T cef42b07900c4b4e8589f284c011c300 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:14.914458 30157 consensus_queue.cc:237] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:14.914592 27652 raft_consensus.cc:2454] T cef42b07900c4b4e8589f284c011c300 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:14.914382 29961 leader_election.cc:302] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:18:14.915134 1129 raft_consensus.cc:2781] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:14.916349 30027 raft_consensus.cc:1261] T a40b618e4cb9426c902c3384dea3bb46 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:14.916343 27652 raft_consensus.cc:1261] T a40b618e4cb9426c902c3384dea3bb46 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:14.916954 1139 consensus_queue.cc:1035] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:14.918149 1139 consensus_queue.cc:1035] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:14.916067 1129 raft_consensus.cc:686] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:18:14.919728 1129 consensus_queue.cc:237] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:14.921034 1139 raft_consensus.cc:2932] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:14.926301 30027 raft_consensus.cc:2932] T a40b618e4cb9426c902c3384dea3bb46 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:14.926920 27389 raft_consensus.cc:2932] T a9e5e67e3fe54491841b33d0d4bd047b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:14.931217 1124 tablet_copy_client.cc:783] T e7064865693149b980ba54fb8e65bc48 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:14.931504 1124 tablet_copy_client.cc:647] T e7064865693149b980ba54fb8e65bc48 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:14.938141 27258 consensus_queue.cc:237] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:14.944031 30159 raft_consensus.cc:2932] T a9e5e67e3fe54491841b33d0d4bd047b P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:14.944926 1124 tablet_copy_client.cc:523] T e7064865693149b980ba54fb8e65bc48 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:14.949538 1124 tablet_bootstrap.cc:492] T e7064865693149b980ba54fb8e65bc48 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:14.952096 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a40b618e4cb9426c902c3384dea3bb46 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:14.955565 30159 raft_consensus.cc:1261] T a9e5e67e3fe54491841b33d0d4bd047b P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:14.956110 930 consensus_queue.cc:1035] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:14.957175 27389 raft_consensus.cc:1261] T a9e5e67e3fe54491841b33d0d4bd047b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:14.957667 930 consensus_queue.cc:1035] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:14.955246 27115 catalog_manager.cc:5484] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:14.962091 27112 catalog_manager.cc:5484] T a40b618e4cb9426c902c3384dea3bb46 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:14.963151 32172 raft_consensus.cc:2932] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:14.963418 30157 raft_consensus.cc:2932] T a9e5e67e3fe54491841b33d0d4bd047b P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:14.966292 30400 tablet_service.cc:1514] Processing DeleteTablet for tablet a40b618e4cb9426c902c3384dea3bb46 with delete_type TABLET_DATA_TOMBSTONED (TS 31a1e33dcdbf42f58ea72ce1c0afb0ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:14.976341 967 tablet_copy_client.cc:783] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:14.976581 967 tablet_copy_client.cc:647] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:14.979720 1133 ts_tablet_manager.cc:1888] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:14.979902 1133 log.cc:1196] T 57505255575249e698bf35929619423d P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/57505255575249e698bf35929619423d
I20240502 08:18:14.982322 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a9e5e67e3fe54491841b33d0d4bd047b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:14.983984 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet 57505255575249e698bf35929619423d (table rebalancer_extra_table_12 [id=e2a08da4858541088d7e3e5a7ae41e45]) successfully deleted
I20240502 08:18:14.984879 1165 ts_tablet_manager.cc:1888] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:14.985179 1165 log.cc:1196] T 16b897307b4c4cbea72f814f64f1e7d7 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/16b897307b4c4cbea72f814f64f1e7d7
I20240502 08:18:14.984858 27390 raft_consensus.cc:2932] T a9e5e67e3fe54491841b33d0d4bd047b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:14.985975 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet 16b897307b4c4cbea72f814f64f1e7d7 (table rebalancer_extra_table_20 [id=aff097c8fa414c8aa9c675914ae4280a]) successfully deleted
I20240502 08:18:14.988283 27650 raft_consensus.cc:2932] T a40b618e4cb9426c902c3384dea3bb46 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:14.992823 1117 raft_consensus.cc:1050] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: attempting to promote NON_VOTER bf8ac71b15f740a0a928911360773442 to VOTER
I20240502 08:18:14.993801 1117 consensus_queue.cc:237] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:14.997071 967 tablet_copy_client.cc:523] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:14.999177 27116 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:14.999402 27116 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:14.999478 27115 catalog_manager.cc:5484] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:15.003595 27116 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet dd2298138c914526ba3825dc3a4ed56c
I20240502 08:18:15.004858 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet dd2298138c914526ba3825dc3a4ed56c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:14 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:18:15.005414 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet dd2298138c914526ba3825dc3a4ed56c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:14 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:15.006630 30401 tablet_service.cc:1514] Processing DeleteTablet for tablet dd2298138c914526ba3825dc3a4ed56c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:14 UTC) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:15.008631 1129 consensus_queue.cc:1035] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:15.015686 967 tablet_bootstrap.cc:492] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:15.016606 30006 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e5e67e3fe54491841b33d0d4bd047b with delete_type TABLET_DATA_TOMBSTONED (TS 1f0bf4bc3c5146539146b825b048bff3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:15.018381 1179 tablet_replica.cc:330] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:15.018414 1133 tablet_replica.cc:330] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:15.020287 1179 raft_consensus.cc:2229] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:15.021073 1179 raft_consensus.cc:2258] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:15.022378 1179 ts_tablet_manager.cc:1875] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:15.028149 27648 raft_consensus.cc:1261] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.031833 1124 tablet_bootstrap.cc:492] T e7064865693149b980ba54fb8e65bc48 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:15.032250 1124 tablet_bootstrap.cc:492] T e7064865693149b980ba54fb8e65bc48 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap complete.
I20240502 08:18:15.035413 390 consensus_queue.cc:1035] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20240502 08:18:15.036784 1124 ts_tablet_manager.cc:1367] T e7064865693149b980ba54fb8e65bc48 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.087s user 0.004s sys 0.005s
I20240502 08:18:15.037483 1124 raft_consensus.cc:348] T e7064865693149b980ba54fb8e65bc48 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:15.041647 30157 raft_consensus.cc:1261] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.041805 1124 raft_consensus.cc:729] T e7064865693149b980ba54fb8e65bc48 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: LEARNER
I20240502 08:18:15.042063 1124 consensus_queue.cc:260] T e7064865693149b980ba54fb8e65bc48 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:15.042251 632 consensus_queue.cc:1035] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.042724 819 consensus_queue.cc:1035] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:15.042850 1124 ts_tablet_manager.cc:1398] T e7064865693149b980ba54fb8e65bc48 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 08:18:15.044925 27516 raft_consensus.cc:1261] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.045473 632 consensus_queue.cc:1035] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.055030 30288 consensus_queue.cc:237] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:15.059927 1179 ts_tablet_manager.cc:1888] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:15.073058 27648 raft_consensus.cc:1203] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 08:18:15.080340 30225 consensus_peers.cc:485] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 -> Peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): Couldn't send request to peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 039b1aea12ba4627882b603336070773. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:15.082288 1187 tablet_replica.cc:330] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:15.082413 30159 raft_consensus.cc:2932] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:15.083288 932 raft_consensus.cc:1050] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 to VOTER
I20240502 08:18:15.084000 932 consensus_queue.cc:237] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:15.089488 30025 raft_consensus.cc:1261] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:15.090248 623 consensus_queue.cc:1035] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:15.090574 27648 raft_consensus.cc:1261] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:15.091336 623 consensus_queue.cc:1035] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:15.093575 27389 raft_consensus.cc:1203] T e5af418a0ae043a387b5030a00b47e91 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:15.107192 27112 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:15.111053 27109 catalog_manager.cc:5484] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:15.118878 27519 raft_consensus.cc:1203] T e7064865693149b980ba54fb8e65bc48 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:15.121040 27368 tablet_service.cc:1467] Processing CreateTablet for tablet 85ea6b6d063b40f4b7e5192653cbff09 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d50d09015f344dc0bf6552a341aaed15]), partition=
I20240502 08:18:15.121384 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ea6b6d063b40f4b7e5192653cbff09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.121490 30006 tablet_service.cc:1467] Processing CreateTablet for tablet 85ea6b6d063b40f4b7e5192653cbff09 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d50d09015f344dc0bf6552a341aaed15]), partition=
I20240502 08:18:15.121830 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ea6b6d063b40f4b7e5192653cbff09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.134057 27519 raft_consensus.cc:2932] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:15.080776 1117 raft_consensus.cc:2932] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:15.135845 1179 log.cc:1196] T a40b618e4cb9426c902c3384dea3bb46 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/a40b618e4cb9426c902c3384dea3bb46
I20240502 08:18:15.116039 30443 tablet_copy_service.cc:342] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Request end of tablet copy session b01e7c8a302d41078fa39cfef665ff59-e7064865693149b980ba54fb8e65bc48 received from {username='slave'} at 127.20.207.3:40171
I20240502 08:18:15.136314 30443 tablet_copy_service.cc:433] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: ending tablet copy session b01e7c8a302d41078fa39cfef665ff59-e7064865693149b980ba54fb8e65bc48 on tablet e7064865693149b980ba54fb8e65bc48 with peer b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:15.121127 30400 tablet_service.cc:1467] Processing CreateTablet for tablet 85ea6b6d063b40f4b7e5192653cbff09 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d50d09015f344dc0bf6552a341aaed15]), partition=
I20240502 08:18:15.137212 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ea6b6d063b40f4b7e5192653cbff09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.137867 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet a40b618e4cb9426c902c3384dea3bb46 (table rebalancer_extra_table_16 [id=98426515f8594fe1951ef5a81f079c16]) successfully deleted
I20240502 08:18:15.118889 1187 raft_consensus.cc:2229] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:15.141360 1187 raft_consensus.cc:2258] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:15.141434 1104 tablet_bootstrap.cc:492] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:15.141775 1104 tablet_bootstrap.cc:492] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap complete.
I20240502 08:18:15.141999 1104 ts_tablet_manager.cc:1367] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.253s user 0.005s sys 0.010s
I20240502 08:18:15.143011 27650 raft_consensus.cc:2932] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:15.145939 1187 ts_tablet_manager.cc:1875] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:15.148726 967 tablet_bootstrap.cc:492] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:15.149130 967 tablet_bootstrap.cc:492] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442: Bootstrap complete.
I20240502 08:18:15.149371 967 ts_tablet_manager.cc:1367] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.134s user 0.004s sys 0.004s
I20240502 08:18:15.150287 1204 tablet_bootstrap.cc:492] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:15.151935 1204 tablet_bootstrap.cc:654] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.153733 1133 raft_consensus.cc:2229] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:15.153818 27255 raft_consensus.cc:1261] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.154085 1133 raft_consensus.cc:2258] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:15.154377 966 raft_consensus.cc:2932] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:15.154981 1133 ts_tablet_manager.cc:1875] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:15.156224 1197 consensus_queue.cc:1035] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.157728 30157 raft_consensus.cc:1261] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.158514 1179 tablet_replica.cc:330] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:15.158707 1179 raft_consensus.cc:2229] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:15.159047 1179 raft_consensus.cc:2258] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:15.159910 1179 ts_tablet_manager.cc:1875] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:15.161902 30028 raft_consensus.cc:2932] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:15.162499 1133 ts_tablet_manager.cc:1888] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:15.162700 1133 log.cc:1196] T a9e5e67e3fe54491841b33d0d4bd047b P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/a9e5e67e3fe54491841b33d0d4bd047b
I20240502 08:18:15.163080 27112 catalog_manager.cc:5484] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:15.163888 1189 ts_tablet_manager.cc:902] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1f0bf4bc3c5146539146b825b048bff3: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:15.164462 485 consensus_queue.cc:1035] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.165553 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet a9e5e67e3fe54491841b33d0d4bd047b (table rebalancer_extra_table_18 [id=2c7cf528be404fbca449af1fcf7d0e23]) successfully deleted
I20240502 08:18:15.165792 1186 tablet_replica.cc:330] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:18:15.165961 1186 raft_consensus.cc:2229] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:15.166188 1186 raft_consensus.cc:2258] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:15.167160 1186 ts_tablet_manager.cc:1875] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:15.170316 30288 raft_consensus.cc:1261] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.171077 1190 consensus_queue.cc:1035] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.176291 1205 tablet_bootstrap.cc:492] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:15.179132 1205 tablet_bootstrap.cc:654] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.185739 967 raft_consensus.cc:348] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:15.186174 967 raft_consensus.cc:729] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: LEARNER
I20240502 08:18:15.186517 967 consensus_queue.cc:260] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:15.187389 967 ts_tablet_manager.cc:1398] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.038s user 0.000s sys 0.004s
I20240502 08:18:15.187649 27648 raft_consensus.cc:2932] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:15.190137 1208 tablet_bootstrap.cc:492] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:15.190197 27258 raft_consensus.cc:1203] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:15.194460 1208 tablet_bootstrap.cc:654] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.194595 1197 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:15.195243 30288 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:15.202625 1189 tablet_copy_client.cc:308] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:18:15.203831 27113 catalog_manager.cc:5484] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:15.205852 1187 ts_tablet_manager.cc:1888] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:15.206153 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet 2dfa509d47e54b0f9ccacc9954981fb6 from peer 1f0bf4bc3c5146539146b825b048bff3 ({username='slave'} at 127.20.207.5:51633)
I20240502 08:18:15.206316 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet 2dfa509d47e54b0f9ccacc9954981fb6 from peer 1f0bf4bc3c5146539146b825b048bff3 at {username='slave'} at 127.20.207.5:51633: session id = 1f0bf4bc3c5146539146b825b048bff3-2dfa509d47e54b0f9ccacc9954981fb6
I20240502 08:18:15.208617 30157 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:15.209154 27278 tablet_copy_source_session.cc:205] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:15.209787 27255 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:15.217639 1189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfa509d47e54b0f9ccacc9954981fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.219259 1186 ts_tablet_manager.cc:1888] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:15.219772 1187 log.cc:1196] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/dd2298138c914526ba3825dc3a4ed56c
I20240502 08:18:15.220280 1187 ts_tablet_manager.cc:1909] T dd2298138c914526ba3825dc3a4ed56c P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:18:15.220327 27519 consensus_queue.cc:237] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:15.225241 1204 tablet_bootstrap.cc:492] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:18:15.225418 1204 ts_tablet_manager.cc:1367] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.000s
I20240502 08:18:15.226246 1204 raft_consensus.cc:348] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:15.226527 1204 raft_consensus.cc:374] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.226649 1204 raft_consensus.cc:729] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.226989 1204 consensus_queue.cc:260] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:15.227954 1204 ts_tablet_manager.cc:1398] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:15.229730 30310 tablet_copy_service.cc:342] P 49c5c8f70e494edd9fcda40cc0af6cc4: Request end of tablet copy session bf8ac71b15f740a0a928911360773442-b1008f7127704fe8b3eb6f9246b0a79c received from {username='slave'} at 127.20.207.4:32967
I20240502 08:18:15.229877 30310 tablet_copy_service.cc:433] P 49c5c8f70e494edd9fcda40cc0af6cc4: ending tablet copy session bf8ac71b15f740a0a928911360773442-b1008f7127704fe8b3eb6f9246b0a79c on tablet b1008f7127704fe8b3eb6f9246b0a79c with peer bf8ac71b15f740a0a928911360773442
I20240502 08:18:15.232358 30288 raft_consensus.cc:1261] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:15.223685 27255 raft_consensus.cc:1261] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:15.142676 1104 raft_consensus.cc:348] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:15.234139 932 consensus_queue.cc:1035] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:15.234254 1104 raft_consensus.cc:729] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: LEARNER
I20240502 08:18:15.234572 1104 consensus_queue.cc:260] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:15.235492 1104 ts_tablet_manager.cc:1398] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.093s user 0.002s sys 0.000s
I20240502 08:18:15.236248 1186 log.cc:1196] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/dd2298138c914526ba3825dc3a4ed56c
I20240502 08:18:15.236234 30048 tablet_copy_service.cc:342] P 1f0bf4bc3c5146539146b825b048bff3: Request end of tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-0dea1ac1af1b45cf9abfd5e5a1d910a1 received from {username='slave'} at 127.20.207.1:49229
I20240502 08:18:15.236382 30048 tablet_copy_service.cc:433] P 1f0bf4bc3c5146539146b825b048bff3: ending tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-0dea1ac1af1b45cf9abfd5e5a1d910a1 on tablet 0dea1ac1af1b45cf9abfd5e5a1d910a1 with peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60
I20240502 08:18:15.236793 1190 consensus_queue.cc:1035] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:15.237676 1186 ts_tablet_manager.cc:1909] T dd2298138c914526ba3825dc3a4ed56c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:18:15.238142 1182 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:15.239792 27255 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:15.244728 30290 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:15.248517 1179 ts_tablet_manager.cc:1888] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:15.250186 1179 log.cc:1196] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/dd2298138c914526ba3825dc3a4ed56c
I20240502 08:18:15.250855 1179 ts_tablet_manager.cc:1909] T dd2298138c914526ba3825dc3a4ed56c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting consensus metadata
I20240502 08:18:15.250924 1205 tablet_bootstrap.cc:492] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:15.251075 1205 ts_tablet_manager.cc:1367] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.000s
I20240502 08:18:15.251811 1205 raft_consensus.cc:348] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:15.252012 1205 raft_consensus.cc:374] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.252103 1205 raft_consensus.cc:729] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.252748 1205 consensus_queue.cc:260] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:15.255793 1205 ts_tablet_manager.cc:1398] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:15.257419 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet dd2298138c914526ba3825dc3a4ed56c (table rebalancer_dynamic_table_new_name [id=c733d5389c7342969dbf12e6bea513f8]) successfully deleted
I20240502 08:18:15.257422 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet dd2298138c914526ba3825dc3a4ed56c (table rebalancer_dynamic_table_new_name [id=c733d5389c7342969dbf12e6bea513f8]) successfully deleted
I20240502 08:18:15.260970 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet de4b822c4aca4f7f92b73cc9279ddec5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:15.261584 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet dd2298138c914526ba3825dc3a4ed56c (table rebalancer_dynamic_table_new_name [id=c733d5389c7342969dbf12e6bea513f8]) successfully deleted
I20240502 08:18:15.264259 27114 catalog_manager.cc:5484] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:15.265588 1189 tablet_copy_client.cc:783] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:15.265877 1189 tablet_copy_client.cc:647] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:15.272315 30290 consensus_queue.cc:237] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:15.276202 1189 tablet_copy_client.cc:523] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:15.278105 1191 raft_consensus.cc:1050] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: attempting to promote NON_VOTER 84cda20738d2452da166e31fc6da57a6 to VOTER
I20240502 08:18:15.278975 1191 consensus_queue.cc:237] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:15.286988 30157 raft_consensus.cc:1261] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:15.287276 1189 tablet_bootstrap.cc:492] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:15.287715 966 consensus_queue.cc:1035] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:15.288897 30226 consensus_peers.cc:485] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bd5657c5cfd4ca4a78cdfdc503b91a8. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:15.290844 27389 raft_consensus.cc:1261] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:15.291460 966 consensus_queue.cc:1035] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:15.293056 1208 tablet_bootstrap.cc:492] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:15.293226 1208 ts_tablet_manager.cc:1367] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.103s user 0.014s sys 0.011s
I20240502 08:18:15.293948 1208 raft_consensus.cc:348] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:15.295467 623 raft_consensus.cc:2932] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:15.296284 30138 tablet_service.cc:1514] Processing DeleteTablet for tablet de4b822c4aca4f7f92b73cc9279ddec5 with delete_type TABLET_DATA_TOMBSTONED (TS 84cda20738d2452da166e31fc6da57a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:15.296103 30159 raft_consensus.cc:2932] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:15.299782 1129 consensus_queue.cc:1035] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:15.304701 27109 catalog_manager.cc:5484] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:15.311805 27390 raft_consensus.cc:2932] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:15.331701 1208 raft_consensus.cc:374] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.332814 1208 raft_consensus.cc:729] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.334080 1208 consensus_queue.cc:260] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:15.340579 1226 tablet_replica.cc:330] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:15.338919 1208 ts_tablet_manager.cc:1398] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20240502 08:18:15.351491 27648 raft_consensus.cc:1261] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.354773 1191 consensus_queue.cc:1035] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.360553 27255 consensus_queue.cc:237] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:15.350920 30157 raft_consensus.cc:1261] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.364638 1026 consensus_queue.cc:1035] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:15.365314 1117 consensus_queue.cc:1035] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.375370 1226 raft_consensus.cc:2229] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:15.376730 1226 raft_consensus.cc:2258] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:15.379344 30288 raft_consensus.cc:1261] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.386415 1117 consensus_queue.cc:1035] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.397956 1226 ts_tablet_manager.cc:1875] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:15.409729 1117 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:15.416741 30289 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:15.430366 27652 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:15.434139 27109 catalog_manager.cc:5484] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 2 to 3, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:15.467221 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Couldn't send request to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 903502e57ab2430383807b0d46a39847. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:15.468304 30288 raft_consensus.cc:1261] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:18:15.469118 32172 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:15.470337 27650 raft_consensus.cc:1261] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:18:15.471901 930 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240502 08:18:15.483352 30157 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:15.487586 390 raft_consensus.cc:1050] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: attempting to promote NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 to VOTER
I20240502 08:18:15.488095 1226 ts_tablet_manager.cc:1888] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:15.488216 1226 log.cc:1196] T de4b822c4aca4f7f92b73cc9279ddec5 P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/de4b822c4aca4f7f92b73cc9279ddec5
I20240502 08:18:15.488379 390 consensus_queue.cc:237] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:15.488801 930 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:15.490607 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet de4b822c4aca4f7f92b73cc9279ddec5 (table rebalancer_extra_table_26 [id=f18ddbbe6f0440debd82be537390727a]) successfully deleted
I20240502 08:18:15.492782 30288 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:15.494776 27652 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:15.500946 27109 catalog_manager.cc:5484] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 7 to 8, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New cstate: current_term: 2 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:15.516963 27113 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:15.526510 30028 raft_consensus.cc:1261] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.527354 632 consensus_queue.cc:1035] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.528379 27255 raft_consensus.cc:1261] T e5af418a0ae043a387b5030a00b47e91 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.529345 1117 consensus_queue.cc:1035] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.532155 27390 raft_consensus.cc:1261] T e5af418a0ae043a387b5030a00b47e91 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.532776 1117 consensus_queue.cc:1035] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.539431 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 5a254106456240879784a4d85d56fca5 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ae0aebcb27544669bbc3f8a4d649bb73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.539911 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a254106456240879784a4d85d56fca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.541486 27500 tablet_service.cc:1467] Processing CreateTablet for tablet 5a254106456240879784a4d85d56fca5 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ae0aebcb27544669bbc3f8a4d649bb73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.541950 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a254106456240879784a4d85d56fca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.544615 30138 tablet_service.cc:1467] Processing CreateTablet for tablet 73548c1dd45f4c31bf3caf89b6d9ebac (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ae0aebcb27544669bbc3f8a4d649bb73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.544818 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 9edac369e0ae4c3c990bbb8dd845b27c (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ae0aebcb27544669bbc3f8a4d649bb73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.545054 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73548c1dd45f4c31bf3caf89b6d9ebac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.545380 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9edac369e0ae4c3c990bbb8dd845b27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.547837 30139 tablet_service.cc:1467] Processing CreateTablet for tablet 9edac369e0ae4c3c990bbb8dd845b27c (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ae0aebcb27544669bbc3f8a4d649bb73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.548168 30139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9edac369e0ae4c3c990bbb8dd845b27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.552145 30006 tablet_service.cc:1467] Processing CreateTablet for tablet 5a254106456240879784a4d85d56fca5 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ae0aebcb27544669bbc3f8a4d649bb73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.552678 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a254106456240879784a4d85d56fca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.556622 30270 tablet_service.cc:1467] Processing CreateTablet for tablet 73548c1dd45f4c31bf3caf89b6d9ebac (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ae0aebcb27544669bbc3f8a4d649bb73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.557024 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73548c1dd45f4c31bf3caf89b6d9ebac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.562155 30269 tablet_service.cc:1467] Processing CreateTablet for tablet 9edac369e0ae4c3c990bbb8dd845b27c (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ae0aebcb27544669bbc3f8a4d649bb73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.562533 30269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9edac369e0ae4c3c990bbb8dd845b27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.562716 30400 tablet_service.cc:1467] Processing CreateTablet for tablet 73548c1dd45f4c31bf3caf89b6d9ebac (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ae0aebcb27544669bbc3f8a4d649bb73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.563136 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73548c1dd45f4c31bf3caf89b6d9ebac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.570361 1208 tablet_bootstrap.cc:492] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:15.572721 1208 tablet_bootstrap.cc:654] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.577217 1026 raft_consensus.cc:482] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:15.577476 1026 raft_consensus.cc:504] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:15.577796 1250 tablet_bootstrap.cc:492] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:15.578547 1026 leader_election.cc:288] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:15.579887 30419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ea6b6d063b40f4b7e5192653cbff09" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:15.580003 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ea6b6d063b40f4b7e5192653cbff09" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:18:15.580147 30419 raft_consensus.cc:2454] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:15.580200 27389 raft_consensus.cc:2454] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:15.580633 29963 leader_election.cc:302] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:15.581063 1026 raft_consensus.cc:2781] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:15.581169 1026 raft_consensus.cc:482] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:15.581288 1026 raft_consensus.cc:3037] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.583988 1026 raft_consensus.cc:504] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:15.584668 30419 consensus_queue.cc:237] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:15.584970 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ea6b6d063b40f4b7e5192653cbff09" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:18:15.585131 27389 raft_consensus.cc:3037] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.587272 27650 raft_consensus.cc:1261] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:15.587307 30157 raft_consensus.cc:1261] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:15.587947 1250 tablet_bootstrap.cc:654] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.588107 390 consensus_queue.cc:1035] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:15.588654 390 consensus_queue.cc:1035] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:15.589545 27389 raft_consensus.cc:2454] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:15.589957 29963 leader_election.cc:302] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:15.590322 819 raft_consensus.cc:2781] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:15.590446 819 raft_consensus.cc:686] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:18:15.590693 819 consensus_queue.cc:237] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:15.592540 1026 leader_election.cc:288] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:15.593609 27650 raft_consensus.cc:1203] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:15.597373 1117 raft_consensus.cc:2932] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:15.602512 30157 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:15.607789 1189 tablet_bootstrap.cc:492] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:15.608700 1250 tablet_bootstrap.cc:492] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:15.608935 1250 ts_tablet_manager.cc:1367] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.004s
I20240502 08:18:15.609879 632 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:15.610854 1256 tablet_bootstrap.cc:492] T 5a254106456240879784a4d85d56fca5 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:15.610839 1250 raft_consensus.cc:348] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:15.611130 1250 raft_consensus.cc:374] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.611259 1250 raft_consensus.cc:729] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.611550 1250 consensus_queue.cc:260] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:15.612285 1250 ts_tablet_manager.cc:1398] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:18:15.612612 1250 tablet_bootstrap.cc:492] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:15.608974 1205 tablet_bootstrap.cc:492] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:15.614869 27650 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:15.615295 1256 tablet_bootstrap.cc:654] T 5a254106456240879784a4d85d56fca5 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.617758 1205 tablet_bootstrap.cc:654] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.618383 27255 raft_consensus.cc:2932] T e5af418a0ae043a387b5030a00b47e91 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:15.619453 390 raft_consensus.cc:1050] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: attempting to promote NON_VOTER b01e7c8a302d41078fa39cfef665ff59 to VOTER
I20240502 08:18:15.620535 390 consensus_queue.cc:237] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:15.622395 1250 tablet_bootstrap.cc:654] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.622884 1189 tablet_bootstrap.cc:492] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap complete.
I20240502 08:18:15.623114 30419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ea6b6d063b40f4b7e5192653cbff09" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:15.623301 30419 raft_consensus.cc:3037] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.623314 626 raft_consensus.cc:1050] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4: attempting to promote NON_VOTER bf8ac71b15f740a0a928911360773442 to VOTER
I20240502 08:18:15.624290 626 consensus_queue.cc:237] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:15.625743 1256 tablet_bootstrap.cc:492] T 5a254106456240879784a4d85d56fca5 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:15.626056 30419 raft_consensus.cc:2454] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:15.626250 1256 ts_tablet_manager.cc:1367] T 5a254106456240879784a4d85d56fca5 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240502 08:18:15.627694 1256 raft_consensus.cc:348] T 5a254106456240879784a4d85d56fca5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:15.628134 1256 raft_consensus.cc:374] T 5a254106456240879784a4d85d56fca5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.628278 1256 raft_consensus.cc:729] T 5a254106456240879784a4d85d56fca5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.628542 1256 consensus_queue.cc:260] T 5a254106456240879784a4d85d56fca5 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:15.629473 1256 ts_tablet_manager.cc:1398] T 5a254106456240879784a4d85d56fca5 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:15.636538 966 raft_consensus.cc:1050] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4: attempting to promote NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 to VOTER
I20240502 08:18:15.637228 966 consensus_queue.cc:237] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:15.636245 30028 raft_consensus.cc:2932] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:15.638437 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4c78e068ec3a451c892d58174b3f837c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:15.642199 1189 ts_tablet_manager.cc:1367] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.355s user 0.093s sys 0.084s
I20240502 08:18:15.643209 1189 raft_consensus.cc:348] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:15.643481 1189 raft_consensus.cc:729] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: LEARNER
I20240502 08:18:15.643734 1189 consensus_queue.cc:260] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:15.648365 27116 catalog_manager.cc:5484] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:15.653913 27113 catalog_manager.cc:5484] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:15.669050 27389 raft_consensus.cc:1261] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.670280 626 consensus_queue.cc:1035] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.669625 27650 raft_consensus.cc:1261] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.671149 626 consensus_queue.cc:1035] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.671957 1252 tablet_bootstrap.cc:492] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:15.672232 30419 raft_consensus.cc:1261] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.672773 623 consensus_queue.cc:1035] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.676998 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c78e068ec3a451c892d58174b3f837c with delete_type TABLET_DATA_TOMBSTONED (TS 49c5c8f70e494edd9fcda40cc0af6cc4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:15.687238 623 raft_consensus.cc:482] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:15.687443 623 raft_consensus.cc:504] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:15.688259 623 leader_election.cc:288] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:15.689105 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a254106456240879784a4d85d56fca5" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:18:15.689318 27519 raft_consensus.cc:2454] T 5a254106456240879784a4d85d56fca5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:15.689906 30226 leader_election.cc:302] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:15.691596 623 raft_consensus.cc:2781] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:15.691699 623 raft_consensus.cc:482] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:15.691795 623 raft_consensus.cc:3037] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.693020 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a254106456240879784a4d85d56fca5" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:15.693789 1189 ts_tablet_manager.cc:1398] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.051s user 0.001s sys 0.001s
I20240502 08:18:15.694213 27258 raft_consensus.cc:1203] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:15.694622 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session 1f0bf4bc3c5146539146b825b048bff3-2dfa509d47e54b0f9ccacc9954981fb6 received from {username='slave'} at 127.20.207.5:51633
I20240502 08:18:15.694849 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session 1f0bf4bc3c5146539146b825b048bff3-2dfa509d47e54b0f9ccacc9954981fb6 on tablet 2dfa509d47e54b0f9ccacc9954981fb6 with peer 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:18:15.695751 623 raft_consensus.cc:504] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:15.696174 1205 tablet_bootstrap.cc:492] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:15.696382 1205 ts_tablet_manager.cc:1367] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.087s user 0.012s sys 0.000s
I20240502 08:18:15.696943 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a254106456240879784a4d85d56fca5" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:18:15.697121 27519 raft_consensus.cc:3037] T 5a254106456240879784a4d85d56fca5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.697144 1205 raft_consensus.cc:348] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:15.697372 1205 raft_consensus.cc:374] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.697440 1205 raft_consensus.cc:729] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.698513 1208 tablet_bootstrap.cc:492] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:15.698724 1208 ts_tablet_manager.cc:1367] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.128s user 0.004s sys 0.007s
I20240502 08:18:15.699729 27519 raft_consensus.cc:2454] T 5a254106456240879784a4d85d56fca5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:15.699687 1208 raft_consensus.cc:348] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:15.699986 1208 raft_consensus.cc:374] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.700078 1208 raft_consensus.cc:729] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.700363 1208 consensus_queue.cc:260] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:15.700807 30226 leader_election.cc:302] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:15.730253 623 leader_election.cc:288] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:15.730434 623 raft_consensus.cc:2781] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:15.730578 623 raft_consensus.cc:686] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:18:15.730803 623 consensus_queue.cc:237] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:15.706804 1104 ts_tablet_manager.cc:902] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Initiating tablet copy from peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:15.718015 27519 raft_consensus.cc:1261] T e7064865693149b980ba54fb8e65bc48 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.730154 27516 raft_consensus.cc:1261] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.719339 30157 raft_consensus.cc:1261] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.721304 1205 consensus_queue.cc:260] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:15.732985 1205 ts_tablet_manager.cc:1398] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.036s user 0.001s sys 0.000s
I20240502 08:18:15.721405 27115 catalog_manager.cc:5484] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:15.732895 626 raft_consensus.cc:2932] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:15.720528 1252 tablet_bootstrap.cc:654] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.730532 30159 raft_consensus.cc:1261] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.745904 1271 consensus_queue.cc:1035] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.746501 1271 consensus_queue.cc:1035] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.751901 27389 raft_consensus.cc:2932] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:15.758616 27115 catalog_manager.cc:5484] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:15.763916 1271 raft_consensus.cc:2932] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:15.766981 27516 raft_consensus.cc:2932] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:15.774570 27115 catalog_manager.cc:5484] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:15.775535 30159 raft_consensus.cc:2932] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:15.712577 1250 tablet_bootstrap.cc:492] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
W20240502 08:18:15.723562 30223 leader_election.cc:341] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:18:15.777747 30223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.207.5:37815, user_credentials={real_user=slave}} blocked reactor thread for 54216.4us
I20240502 08:18:15.778792 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a254106456240879784a4d85d56fca5" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:15.779012 30025 raft_consensus.cc:3037] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.779863 1250 ts_tablet_manager.cc:1367] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.167s user 0.005s sys 0.000s
I20240502 08:18:15.780675 1250 raft_consensus.cc:348] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:15.780872 1250 raft_consensus.cc:374] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.780938 1250 raft_consensus.cc:729] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.781669 30025 raft_consensus.cc:2454] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:15.713027 1255 tablet_bootstrap.cc:492] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:15.718254 27258 raft_consensus.cc:1261] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.724587 27255 raft_consensus.cc:1261] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:15.789114 1271 consensus_queue.cc:1035] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.726886 1267 tablet_replica.cc:330] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:15.808173 1267 raft_consensus.cc:2229] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:15.808524 1267 raft_consensus.cc:2258] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:15.809470 1267 ts_tablet_manager.cc:1875] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:15.811452 27113 catalog_manager.cc:5484] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7). New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:15.816738 30288 consensus_queue.cc:237] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:15.733191 1117 consensus_queue.cc:1035] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.818125 1208 ts_tablet_manager.cc:1398] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.119s user 0.003s sys 0.000s
I20240502 08:18:15.779245 30419 raft_consensus.cc:2932] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:15.819891 1255 tablet_bootstrap.cc:654] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.821784 632 consensus_queue.cc:1035] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.781744 27650 raft_consensus.cc:2932] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:15.823092 1246 consensus_queue.cc:1035] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:15.826701 1267 ts_tablet_manager.cc:1888] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:15.826939 27389 raft_consensus.cc:1261] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:15.826895 1250 consensus_queue.cc:260] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:15.833513 966 consensus_queue.cc:1035] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:15.836086 1250 ts_tablet_manager.cc:1398] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.056s user 0.002s sys 0.000s
I20240502 08:18:15.836376 1250 tablet_bootstrap.cc:492] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:15.836663 27255 raft_consensus.cc:2932] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:15.837968 1250 tablet_bootstrap.cc:654] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.838474 1271 raft_consensus.cc:2932] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:15.840227 27389 raft_consensus.cc:2932] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:15.840950 1250 tablet_bootstrap.cc:492] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:15.845381 1250 ts_tablet_manager.cc:1367] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 08:18:15.841120 1267 log.cc:1196] T 4c78e068ec3a451c892d58174b3f837c P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/4c78e068ec3a451c892d58174b3f837c
I20240502 08:18:15.846910 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet 4c78e068ec3a451c892d58174b3f837c (table rebalancer_extra_table_36 [id=0f9f00e51ec94f5f9d43f80dbb78ff7a]) successfully deleted
I20240502 08:18:15.849388 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f1f416f090284b8d92d53662cc7a3b97 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:15.850149 1250 raft_consensus.cc:348] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:15.851038 1250 raft_consensus.cc:374] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.851290 1250 raft_consensus.cc:729] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.851760 1250 consensus_queue.cc:260] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:15.853049 1250 ts_tablet_manager.cc:1398] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:18:15.853622 27652 raft_consensus.cc:1261] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:15.855106 1271 consensus_queue.cc:1035] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:15.856276 27648 raft_consensus.cc:2932] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:15.861335 1287 ts_tablet_manager.cc:902] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P b01e7c8a302d41078fa39cfef665ff59: Initiating tablet copy from peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:15.870844 27389 raft_consensus.cc:2932] T e5af418a0ae043a387b5030a00b47e91 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:15.877399 27114 catalog_manager.cc:5484] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:15.879511 1270 raft_consensus.cc:482] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:15.879668 1270 raft_consensus.cc:504] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:15.880280 1270 leader_election.cc:288] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:15.881381 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9edac369e0ae4c3c990bbb8dd845b27c" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
W20240502 08:18:15.883620 30225 leader_election.cc:341] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:15.888085 30419 consensus_queue.cc:237] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:15.889217 30400 tablet_service.cc:1514] Processing DeleteTablet for tablet f1f416f090284b8d92d53662cc7a3b97 with delete_type TABLET_DATA_TOMBSTONED (TS 31a1e33dcdbf42f58ea72ce1c0afb0ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:15.890245 27258 raft_consensus.cc:1261] T e5af418a0ae043a387b5030a00b47e91 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:15.890985 1246 consensus_queue.cc:1035] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:15.892830 27388 raft_consensus.cc:1261] T e5af418a0ae043a387b5030a00b47e91 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:15.894807 1117 consensus_queue.cc:1035] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:15.895635 1117 raft_consensus.cc:2932] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:15.902446 1117 raft_consensus.cc:482] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:15.902626 1117 raft_consensus.cc:504] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:15.903726 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e5af418a0ae043a387b5030a00b47e91 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:15.904464 1117 leader_election.cc:288] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:15.904603 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73548c1dd45f4c31bf3caf89b6d9ebac" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:15.904803 30288 raft_consensus.cc:2454] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:15.905306 30355 leader_election.cc:302] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:18:15.906427 1117 raft_consensus.cc:2781] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:15.906633 1117 raft_consensus.cc:482] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:15.906795 1117 raft_consensus.cc:3037] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.914602 27255 raft_consensus.cc:2932] T e5af418a0ae043a387b5030a00b47e91 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:15.913305 27116 catalog_manager.cc:5484] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:15.917799 27115 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:15.925563 30006 tablet_service.cc:1514] Processing DeleteTablet for tablet e5af418a0ae043a387b5030a00b47e91 with delete_type TABLET_DATA_TOMBSTONED (TS 1f0bf4bc3c5146539146b825b048bff3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:15.929772 1117 raft_consensus.cc:504] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:15.931495 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73548c1dd45f4c31bf3caf89b6d9ebac" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:15.931789 30288 raft_consensus.cc:3037] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.939427 30288 raft_consensus.cc:2454] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:15.939890 30355 leader_election.cc:302] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:18:15.940212 1191 raft_consensus.cc:2781] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:15.940426 1191 raft_consensus.cc:686] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:18:15.940698 1191 consensus_queue.cc:237] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:15.944119 1255 tablet_bootstrap.cc:492] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:15.945137 1255 ts_tablet_manager.cc:1367] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.232s user 0.009s sys 0.010s
I20240502 08:18:15.946419 1255 raft_consensus.cc:348] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:15.948758 30156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73548c1dd45f4c31bf3caf89b6d9ebac" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:15.949165 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73548c1dd45f4c31bf3caf89b6d9ebac" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:15.952548 1255 raft_consensus.cc:374] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.952652 1255 raft_consensus.cc:729] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.955003 27648 consensus_queue.cc:237] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:15.955837 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9edac369e0ae4c3c990bbb8dd845b27c" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:15.952875 1255 consensus_queue.cc:260] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
W20240502 08:18:15.956741 30358 leader_election.cc:341] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:18:15.957123 30358 leader_election.cc:341] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:15.957245 1255 ts_tablet_manager.cc:1398] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
W20240502 08:18:15.959342 30226 leader_election.cc:341] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:15.959503 30226 leader_election.cc:302] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4; no voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:15.962456 1271 raft_consensus.cc:2726] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:15.964232 27114 catalog_manager.cc:5484] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:15.965004 1252 tablet_bootstrap.cc:492] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:15.965235 1252 ts_tablet_manager.cc:1367] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.293s user 0.021s sys 0.027s
I20240502 08:18:15.967191 632 raft_consensus.cc:2932] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:15.968060 1252 raft_consensus.cc:348] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:15.969686 1252 raft_consensus.cc:374] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.969789 1252 raft_consensus.cc:729] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.970048 1252 consensus_queue.cc:260] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:15.972259 30157 raft_consensus.cc:2932] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:15.975627 1139 raft_consensus.cc:482] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:15.975850 1139 raft_consensus.cc:504] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:15.976575 1139 leader_election.cc:288] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:15.977299 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73548c1dd45f4c31bf3caf89b6d9ebac" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:15.977524 30288 raft_consensus.cc:2379] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84cda20738d2452da166e31fc6da57a6 in current term 1: Already voted for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in this term.
I20240502 08:18:15.977648 1117 leader_election.cc:288] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:15.977808 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73548c1dd45f4c31bf3caf89b6d9ebac" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:15.980244 1291 tablet_replica.cc:330] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:15.980530 1291 raft_consensus.cc:2229] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:15.980913 1291 raft_consensus.cc:2258] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:15.984762 1296 tablet_replica.cc:330] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:15.984421 27258 raft_consensus.cc:2932] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:15.984992 1296 raft_consensus.cc:2229] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:15.985189 1296 raft_consensus.cc:2258] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:15.985157 30093 leader_election.cc:302] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:15.985637 1139 raft_consensus.cc:3037] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.988684 1296 ts_tablet_manager.cc:1875] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:15.988711 1139 raft_consensus.cc:2726] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:15.989301 1104 tablet_copy_client.cc:308] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.7:40343
I20240502 08:18:15.990767 30310 tablet_copy_service.cc:140] P 49c5c8f70e494edd9fcda40cc0af6cc4: Received BeginTabletCopySession request for tablet 039b1aea12ba4627882b603336070773 from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 ({username='slave'} at 127.20.207.1:49483)
I20240502 08:18:15.990957 30310 tablet_copy_service.cc:161] P 49c5c8f70e494edd9fcda40cc0af6cc4: Beginning new tablet copy session on tablet 039b1aea12ba4627882b603336070773 from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 at {username='slave'} at 127.20.207.1:49483: session id = 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-039b1aea12ba4627882b603336070773
I20240502 08:18:15.993294 27113 catalog_manager.cc:5484] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:15.993880 30310 tablet_copy_source_session.cc:205] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:15.975415 1252 ts_tablet_manager.cc:1398] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 08:18:15.998272 1104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039b1aea12ba4627882b603336070773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.999650 1252 tablet_bootstrap.cc:492] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:16.002638 1296 ts_tablet_manager.cc:1888] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:15.997674 27516 raft_consensus.cc:2932] T e7064865693149b980ba54fb8e65bc48 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:16.004518 1296 log.cc:1196] T e5af418a0ae043a387b5030a00b47e91 P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/e5af418a0ae043a387b5030a00b47e91
I20240502 08:18:16.005391 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet e5af418a0ae043a387b5030a00b47e91 (table rebalancer_extra_table_30 [id=8fad8146798f48ce9901b24b7f1944b1]) successfully deleted
I20240502 08:18:16.004369 1287 tablet_copy_client.cc:308] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.7:40343
I20240502 08:18:16.010231 30310 tablet_copy_service.cc:140] P 49c5c8f70e494edd9fcda40cc0af6cc4: Received BeginTabletCopySession request for tablet 9bd5657c5cfd4ca4a78cdfdc503b91a8 from peer b01e7c8a302d41078fa39cfef665ff59 ({username='slave'} at 127.20.207.3:49869)
I20240502 08:18:16.010459 30310 tablet_copy_service.cc:161] P 49c5c8f70e494edd9fcda40cc0af6cc4: Beginning new tablet copy session on tablet 9bd5657c5cfd4ca4a78cdfdc503b91a8 from peer b01e7c8a302d41078fa39cfef665ff59 at {username='slave'} at 127.20.207.3:49869: session id = b01e7c8a302d41078fa39cfef665ff59-9bd5657c5cfd4ca4a78cdfdc503b91a8
I20240502 08:18:16.012301 27390 raft_consensus.cc:1261] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:16.012761 1086 consensus_queue.cc:1035] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:16.013965 30310 tablet_copy_source_session.cc:205] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:16.014076 1104 tablet_copy_client.cc:783] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:16.014350 1104 tablet_copy_client.cc:647] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:16.015393 1287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd5657c5cfd4ca4a78cdfdc503b91a8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:18:16.017328 27587 consensus_peers.cc:485] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 -> Peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Couldn't send request to peer 84cda20738d2452da166e31fc6da57a6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33aee3df5eb242ef9cc056d25f6d6a82. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:16.022395 27390 raft_consensus.cc:1261] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:15.985692 1295 ts_tablet_manager.cc:902] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:15.995054 30419 raft_consensus.cc:1261] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:16.022169 30423 raft_consensus.cc:1261] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:16.023415 1287 tablet_copy_client.cc:783] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:16.023725 1287 tablet_copy_client.cc:647] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:16.023916 1129 consensus_queue.cc:1035] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.029623 1104 tablet_copy_client.cc:523] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:16.033097 1104 tablet_bootstrap.cc:492] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:16.034116 1252 tablet_bootstrap.cc:654] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.035604 1287 tablet_copy_client.cc:523] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:16.038882 1287 tablet_bootstrap.cc:492] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:16.055320 1252 tablet_bootstrap.cc:492] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:16.055666 1252 ts_tablet_manager.cc:1367] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.002s
I20240502 08:18:16.056552 1252 raft_consensus.cc:348] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:16.056881 1252 raft_consensus.cc:374] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.057016 1252 raft_consensus.cc:729] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.057394 1252 consensus_queue.cc:260] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:16.060863 1252 ts_tablet_manager.cc:1398] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:16.061694 30025 raft_consensus.cc:1203] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:18:16.070309 1291 ts_tablet_manager.cc:1875] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:16.076130 1287 tablet_bootstrap.cc:492] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:16.078922 27389 raft_consensus.cc:2932] T e5af418a0ae043a387b5030a00b47e91 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:16.079770 1287 tablet_bootstrap.cc:492] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap complete.
I20240502 08:18:16.080070 1287 ts_tablet_manager.cc:1367] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.000s
I20240502 08:18:16.080874 1287 raft_consensus.cc:348] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:16.081229 1287 raft_consensus.cc:729] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: LEARNER
I20240502 08:18:16.081488 1287 consensus_queue.cc:260] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:16.082298 1287 ts_tablet_manager.cc:1398] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:16.084697 1129 consensus_queue.cc:1035] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.087181 1306 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:16.091001 1309 consensus_queue.cc:1035] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:16.092790 27388 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:16.096225 1300 raft_consensus.cc:482] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:16.096379 1300 raft_consensus.cc:504] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:16.097157 1300 leader_election.cc:288] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:16.097378 30310 tablet_copy_service.cc:342] P 49c5c8f70e494edd9fcda40cc0af6cc4: Request end of tablet copy session b01e7c8a302d41078fa39cfef665ff59-9bd5657c5cfd4ca4a78cdfdc503b91a8 received from {username='slave'} at 127.20.207.3:49869
I20240502 08:18:16.097506 30310 tablet_copy_service.cc:433] P 49c5c8f70e494edd9fcda40cc0af6cc4: ending tablet copy session b01e7c8a302d41078fa39cfef665ff59-9bd5657c5cfd4ca4a78cdfdc503b91a8 on tablet 9bd5657c5cfd4ca4a78cdfdc503b91a8 with peer b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:16.099455 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9edac369e0ae4c3c990bbb8dd845b27c" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:16.099690 27258 raft_consensus.cc:2454] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:16.100232 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9edac369e0ae4c3c990bbb8dd845b27c" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:16.100486 30288 raft_consensus.cc:2454] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:16.101058 30093 leader_election.cc:302] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:16.101866 1300 raft_consensus.cc:2781] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:16.101986 1300 raft_consensus.cc:482] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:16.102105 1300 raft_consensus.cc:3037] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.103360 1270 raft_consensus.cc:1050] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4: attempting to promote NON_VOTER bf8ac71b15f740a0a928911360773442 to VOTER
I20240502 08:18:16.104120 1270 consensus_queue.cc:237] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:16.106858 1300 raft_consensus.cc:504] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:16.108215 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9edac369e0ae4c3c990bbb8dd845b27c" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:16.108418 27258 raft_consensus.cc:3037] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.115432 27258 raft_consensus.cc:2454] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:16.115936 30095 leader_election.cc:302] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:16.116402 1139 raft_consensus.cc:2781] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:16.116535 1139 raft_consensus.cc:686] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:16.116843 1139 consensus_queue.cc:237] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:16.119426 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9edac369e0ae4c3c990bbb8dd845b27c" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:16.119644 30288 raft_consensus.cc:3037] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.122972 27648 raft_consensus.cc:1261] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:16.123512 966 consensus_queue.cc:1035] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:16.132542 30423 raft_consensus.cc:1261] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:16.133143 966 consensus_queue.cc:1035] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:16.136626 1304 mvcc.cc:205] Tried to move back new op lower bound from 7023156821974315008 to 7023156820341772288. Current Snapshot: MvccSnapshot[applied={T|T < 7023156821974315008}]
I20240502 08:18:16.145231 27113 catalog_manager.cc:5484] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:16.148072 27388 raft_consensus.cc:1261] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:16.148631 27109 catalog_manager.cc:5576] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619) does not have the latest schema for tablet 85ea6b6d063b40f4b7e5192653cbff09 (table rebalancer_dynamic_table [id=d50d09015f344dc0bf6552a341aaed15]). Expected version 1 got 0
I20240502 08:18:16.150131 1270 consensus_queue.cc:1035] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:16.151598 30422 consensus_queue.cc:237] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:16.153637 27516 raft_consensus.cc:1261] T e7064865693149b980ba54fb8e65bc48 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:16.154208 1246 consensus_queue.cc:1035] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:16.154315 27258 raft_consensus.cc:1261] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:16.155155 632 consensus_queue.cc:1035] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:16.156544 632 raft_consensus.cc:2932] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:16.159112 27516 raft_consensus.cc:2932] T e7064865693149b980ba54fb8e65bc48 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:16.161345 1319 tablet.cc:1722] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:16.165853 1104 tablet_bootstrap.cc:492] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:16.168246 1319 tablet.cc:1620] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:16.168388 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e7064865693149b980ba54fb8e65bc48 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:16.168150 966 raft_consensus.cc:2932] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:16.161037 1322 tablet.cc:1722] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:16.168677 30419 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:16.169262 1322 tablet.cc:1620] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:16.169920 1104 tablet_bootstrap.cc:492] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap complete.
I20240502 08:18:16.170218 1104 ts_tablet_manager.cc:1367] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.137s user 0.011s sys 0.006s
I20240502 08:18:16.170779 30288 raft_consensus.cc:2454] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:16.170905 1104 raft_consensus.cc:348] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:16.171120 1104 raft_consensus.cc:729] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: LEARNER
I20240502 08:18:16.171326 1104 consensus_queue.cc:260] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:16.171880 1104 ts_tablet_manager.cc:1398] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:16.173846 30310 tablet_copy_service.cc:342] P 49c5c8f70e494edd9fcda40cc0af6cc4: Request end of tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-039b1aea12ba4627882b603336070773 received from {username='slave'} at 127.20.207.1:49483
I20240502 08:18:16.173480 27650 raft_consensus.cc:2932] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:16.173974 30310 tablet_copy_service.cc:433] P 49c5c8f70e494edd9fcda40cc0af6cc4: ending tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-039b1aea12ba4627882b603336070773 on tablet 039b1aea12ba4627882b603336070773 with peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60
I20240502 08:18:16.174544 30423 raft_consensus.cc:2932] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:16.180267 1300 leader_election.cc:288] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:16.183773 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=d50d09015f344dc0bf6552a341aaed15] alter complete (version 1)
I20240502 08:18:16.184897 27258 raft_consensus.cc:2932] T e7064865693149b980ba54fb8e65bc48 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:16.185072 27114 catalog_manager.cc:5484] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:16.188817 27255 raft_consensus.cc:1203] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:16.193989 966 consensus_queue.cc:1035] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.196388 27110 catalog_manager.cc:5484] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:16.202160 30138 tablet_service.cc:1514] Processing DeleteTablet for tablet e7064865693149b980ba54fb8e65bc48 with delete_type TABLET_DATA_TOMBSTONED (TS 84cda20738d2452da166e31fc6da57a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:16.211967 1026 raft_consensus.cc:1050] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3: attempting to promote NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 to VOTER
I20240502 08:18:16.212775 27115 catalog_manager.cc:5576] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291) does not have the latest schema for tablet 85ea6b6d063b40f4b7e5192653cbff09 (table rebalancer_dynamic_table [id=d50d09015f344dc0bf6552a341aaed15]). Expected version 1 got 0
I20240502 08:18:16.212797 1026 consensus_queue.cc:237] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:16.213436 1291 ts_tablet_manager.cc:1888] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:16.213560 1291 log.cc:1196] T f1f416f090284b8d92d53662cc7a3b97 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/f1f416f090284b8d92d53662cc7a3b97
I20240502 08:18:16.214702 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet f1f416f090284b8d92d53662cc7a3b97 (table rebalancer_extra_table_34 [id=5ebae50d574942c6976b1ce3e50307d5]) successfully deleted
I20240502 08:18:16.218633 27255 raft_consensus.cc:1261] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:16.219657 819 consensus_queue.cc:1035] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:16.220765 30289 raft_consensus.cc:1261] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:16.221884 819 consensus_queue.cc:1035] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:16.223116 30419 raft_consensus.cc:1261] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:16.223659 819 consensus_queue.cc:1035] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:16.246273 1129 raft_consensus.cc:2932] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:16.250353 27255 raft_consensus.cc:2932] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:16.256916 30289 raft_consensus.cc:2932] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:16.267882 27113 catalog_manager.cc:5484] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:16.270479 30423 raft_consensus.cc:2932] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:16.273487 1335 tablet.cc:1722] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:16.279044 27518 raft_consensus.cc:1203] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:16.279911 1335 tablet.cc:1620] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1: MemRowSet was empty: no flush needed.
I20240502 08:18:16.281102 27115 catalog_manager.cc:5576] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291) does not have the latest schema for tablet 85ea6b6d063b40f4b7e5192653cbff09 (table rebalancer_dynamic_table [id=d50d09015f344dc0bf6552a341aaed15]). Expected version 1 got 0
I20240502 08:18:16.284735 30289 consensus_queue.cc:237] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:16.286212 1327 tablet_replica.cc:330] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:16.286442 1327 raft_consensus.cc:2229] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:16.286741 1327 raft_consensus.cc:2258] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:16.287621 27255 raft_consensus.cc:1261] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:16.287664 1327 ts_tablet_manager.cc:1875] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:16.287662 30159 raft_consensus.cc:1261] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:16.288276 1270 consensus_queue.cc:1035] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:16.289939 1271 consensus_queue.cc:1035] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:16.293587 1271 raft_consensus.cc:2932] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:16.297888 30159 raft_consensus.cc:2932] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:16.305842 27389 raft_consensus.cc:2932] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:16.308784 27255 raft_consensus.cc:2932] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:16.314496 27116 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:16.316444 1327 ts_tablet_manager.cc:1888] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:16.316566 1327 log.cc:1196] T e7064865693149b980ba54fb8e65bc48 P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/e7064865693149b980ba54fb8e65bc48
I20240502 08:18:16.317817 1295 tablet_copy_client.cc:308] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:18:16.318887 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet 903502e57ab2430383807b0d46a39847 from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad ({username='slave'} at 127.20.207.8:40413)
I20240502 08:18:16.319016 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet 903502e57ab2430383807b0d46a39847 from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad at {username='slave'} at 127.20.207.8:40413: session id = 31a1e33dcdbf42f58ea72ce1c0afb0ad-903502e57ab2430383807b0d46a39847
I20240502 08:18:16.321393 27278 tablet_copy_source_session.cc:205] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:16.321409 1271 consensus_queue.cc:1035] T 5a254106456240879784a4d85d56fca5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.323410 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet e7064865693149b980ba54fb8e65bc48 (table rebalancer_extra_table_16 [id=98426515f8594fe1951ef5a81f079c16]) successfully deleted
I20240502 08:18:16.325249 1295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903502e57ab2430383807b0d46a39847. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.327440 632 consensus_queue.cc:1035] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.334928 27110 catalog_manager.cc:5484] T 757f65cd77bf49d882c2545673282e03 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 3 to 4, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:16.341892 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet 757f65cd77bf49d882c2545673282e03 with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:16.357317 1295 tablet_copy_client.cc:783] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:16.357573 1295 tablet_copy_client.cc:647] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:16.360201 1295 tablet_copy_client.cc:523] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:16.360112 27388 consensus_queue.cc:237] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:16.365707 1295 tablet_bootstrap.cc:492] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:16.370291 30025 raft_consensus.cc:1261] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:16.370775 27326 consensus_peers.cc:485] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d309785afd8467dab3418bb63c95030. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:16.371330 30419 raft_consensus.cc:1261] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:16.371745 1101 consensus_queue.cc:1035] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:16.373432 1101 consensus_queue.cc:1035] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:16.378202 1117 consensus_queue.cc:1035] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.406414 1294 raft_consensus.cc:2932] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:16.410513 27115 catalog_manager.cc:5576] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291) does not have the latest schema for tablet 85ea6b6d063b40f4b7e5192653cbff09 (table rebalancer_dynamic_table [id=d50d09015f344dc0bf6552a341aaed15]). Expected version 2 got 1
I20240502 08:18:16.410980 30419 raft_consensus.cc:2932] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:16.417466 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 757f65cd77bf49d882c2545673282e03 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:16.418660 27116 catalog_manager.cc:5484] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:16.419533 30289 consensus_queue.cc:237] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:16.421294 27648 raft_consensus.cc:1261] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:16.421936 1271 consensus_queue.cc:1035] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:16.422956 1319 tablet.cc:1722] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:16.423399 1319 tablet.cc:1620] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:16.423503 1335 tablet.cc:1722] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:16.423899 1335 tablet.cc:1620] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1: MemRowSet was empty: no flush needed.
I20240502 08:18:16.424789 27390 raft_consensus.cc:1261] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:16.429148 1270 consensus_queue.cc:1035] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:16.429360 27109 catalog_manager.cc:5484] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:16.429867 1270 raft_consensus.cc:2932] T b1008f7127704fe8b3eb6f9246b0a79c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:16.430843 30025 raft_consensus.cc:2932] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:16.435092 27648 raft_consensus.cc:2932] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:16.440635 27113 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet 85ea6b6d063b40f4b7e5192653cbff09 (table rebalancer_dynamic_table [id=d50d09015f344dc0bf6552a341aaed15]). Expected version 2 got 1
I20240502 08:18:16.442569 1322 tablet.cc:1722] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:16.443042 1322 tablet.cc:1620] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: MemRowSet was empty: no flush needed.
I20240502 08:18:16.447218 27390 raft_consensus.cc:2932] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:16.451403 30027 consensus_queue.cc:237] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:16.454164 30419 raft_consensus.cc:1261] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:16.454700 1026 consensus_queue.cc:1035] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:16.457573 27258 raft_consensus.cc:1261] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:16.458329 1129 consensus_queue.cc:1035] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:16.461186 1129 raft_consensus.cc:2932] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:16.467294 27255 raft_consensus.cc:2932] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:16.480096 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0dea1ac1af1b45cf9abfd5e5a1d910a1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:16.480345 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b1008f7127704fe8b3eb6f9246b0a79c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:16.483613 27116 catalog_manager.cc:5484] T b1008f7127704fe8b3eb6f9246b0a79c P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:16.488389 1352 ts_tablet_manager.cc:902] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6: Initiating tablet copy from peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:16.489223 1344 tablet_replica.cc:330] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:16.489445 1344 raft_consensus.cc:2229] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:16.489701 1344 raft_consensus.cc:2258] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:16.490623 1344 ts_tablet_manager.cc:1875] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:16.494292 30423 raft_consensus.cc:2932] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:16.495932 27257 consensus_queue.cc:237] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:16.504736 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=d50d09015f344dc0bf6552a341aaed15] alter complete (version 2)
W20240502 08:18:16.509083 27191 consensus_peers.cc:485] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a793eaaeba89458baf8af7caf3f24c70. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:16.517132 1344 ts_tablet_manager.cc:1888] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:16.521739 27113 catalog_manager.cc:5484] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:16.522951 1344 log.cc:1196] T 757f65cd77bf49d882c2545673282e03 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/757f65cd77bf49d882c2545673282e03
I20240502 08:18:16.523118 1139 consensus_queue.cc:1035] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.523744 1352 tablet_copy_client.cc:308] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.4:33309
I20240502 08:18:16.524958 27671 tablet_copy_service.cc:140] P bf8ac71b15f740a0a928911360773442: Received BeginTabletCopySession request for tablet 33aee3df5eb242ef9cc056d25f6d6a82 from peer 84cda20738d2452da166e31fc6da57a6 ({username='slave'} at 127.20.207.6:47641)
I20240502 08:18:16.525111 27671 tablet_copy_service.cc:161] P bf8ac71b15f740a0a928911360773442: Beginning new tablet copy session on tablet 33aee3df5eb242ef9cc056d25f6d6a82 from peer 84cda20738d2452da166e31fc6da57a6 at {username='slave'} at 127.20.207.6:47641: session id = 84cda20738d2452da166e31fc6da57a6-33aee3df5eb242ef9cc056d25f6d6a82
I20240502 08:18:16.525441 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 757f65cd77bf49d882c2545673282e03 (table rebalancer_extra_table_32 [id=3c7f8803df8a4ae885409a0c0a114f1f]) successfully deleted
I20240502 08:18:16.528240 27671 tablet_copy_source_session.cc:205] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:16.530230 1352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33aee3df5eb242ef9cc056d25f6d6a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.540983 30400 tablet_service.cc:1514] Processing DeleteTablet for tablet b1008f7127704fe8b3eb6f9246b0a79c with delete_type TABLET_DATA_TOMBSTONED (TS 31a1e33dcdbf42f58ea72ce1c0afb0ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:16.541621 1291 tablet_replica.cc:330] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:16.541898 1291 raft_consensus.cc:2229] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:16.542213 1291 raft_consensus.cc:2258] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:16.543115 1291 ts_tablet_manager.cc:1875] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:16.544826 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dea1ac1af1b45cf9abfd5e5a1d910a1 with delete_type TABLET_DATA_TOMBSTONED (TS 49c5c8f70e494edd9fcda40cc0af6cc4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:16.549496 1291 ts_tablet_manager.cc:1888] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:16.549683 1291 log.cc:1196] T b1008f7127704fe8b3eb6f9246b0a79c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/b1008f7127704fe8b3eb6f9246b0a79c
I20240502 08:18:16.551241 1300 consensus_queue.cc:1035] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.554838 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet b1008f7127704fe8b3eb6f9246b0a79c (table rebalancer_extra_table_14 [id=95b099178a3c41f7b5f6e2f30dfe9f42]) successfully deleted
I20240502 08:18:16.555027 1352 tablet_copy_client.cc:783] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:16.555850 1352 tablet_copy_client.cc:647] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:16.565176 1366 tablet_replica.cc:330] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:16.568042 1366 raft_consensus.cc:2229] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:16.568336 1366 raft_consensus.cc:2258] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:16.568933 1352 tablet_copy_client.cc:523] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:16.569288 1366 ts_tablet_manager.cc:1875] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:16.575932 1366 ts_tablet_manager.cc:1888] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:16.577726 1352 tablet_bootstrap.cc:492] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:16.581401 27390 raft_consensus.cc:1261] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:16.581573 30159 raft_consensus.cc:1261] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:16.581977 32172 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:16.583444 837 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:16.595005 1362 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:16.595542 1366 log.cc:1196] T 0dea1ac1af1b45cf9abfd5e5a1d910a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/0dea1ac1af1b45cf9abfd5e5a1d910a1
I20240502 08:18:16.595568 27388 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:16.596416 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet 0dea1ac1af1b45cf9abfd5e5a1d910a1 (table rebalancer_extra_table_12 [id=e2a08da4858541088d7e3e5a7ae41e45]) successfully deleted
I20240502 08:18:16.603129 27109 catalog_manager.cc:5484] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 6 to 7, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:16.609371 30159 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:16.618810 27113 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:16.631489 1319 tablet.cc:1722] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:16.635116 1322 tablet.cc:1722] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:16.643613 32172 raft_consensus.cc:1050] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 to VOTER
I20240502 08:18:16.644706 32172 consensus_queue.cc:237] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:16.649868 27113 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=d50d09015f344dc0bf6552a341aaed15] alter complete (version 3)
I20240502 08:18:16.652177 1271 raft_consensus.cc:1050] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4: attempting to promote NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 to VOTER
I20240502 08:18:16.652760 1295 tablet_bootstrap.cc:492] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:16.652978 1271 consensus_queue.cc:237] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:16.653250 1295 tablet_bootstrap.cc:492] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap complete.
I20240502 08:18:16.653512 1295 ts_tablet_manager.cc:1367] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.288s user 0.062s sys 0.062s
I20240502 08:18:16.653653 1335 tablet.cc:1722] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:16.654217 1295 raft_consensus.cc:348] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:16.654472 1295 raft_consensus.cc:729] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: LEARNER
I20240502 08:18:16.654649 1295 consensus_queue.cc:260] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:16.659297 1295 ts_tablet_manager.cc:1398] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:18:16.663302 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-903502e57ab2430383807b0d46a39847 received from {username='slave'} at 127.20.207.8:40413
I20240502 08:18:16.663404 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-903502e57ab2430383807b0d46a39847 on tablet 903502e57ab2430383807b0d46a39847 with peer 31a1e33dcdbf42f58ea72ce1c0afb0ad
I20240502 08:18:16.678787 1352 tablet_bootstrap.cc:492] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:16.687556 27648 raft_consensus.cc:1261] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:16.688102 1270 consensus_queue.cc:1035] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:16.690903 27255 raft_consensus.cc:1261] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:16.691670 1270 consensus_queue.cc:1035] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:16.694434 30025 raft_consensus.cc:1261] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:16.694758 30027 raft_consensus.cc:1261] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:16.695290 1270 consensus_queue.cc:1035] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:16.695405 1362 consensus_queue.cc:1035] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:16.699069 27388 raft_consensus.cc:1261] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:16.699658 837 consensus_queue.cc:1035] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:16.701594 1270 raft_consensus.cc:2932] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:16.702134 30288 raft_consensus.cc:1261] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:16.702737 30025 raft_consensus.cc:2932] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:16.704778 1362 consensus_queue.cc:1035] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:16.705559 1352 tablet_bootstrap.cc:492] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6: Bootstrap complete.
I20240502 08:18:16.705863 1352 ts_tablet_manager.cc:1367] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.128s user 0.050s sys 0.003s
I20240502 08:18:16.706810 1352 raft_consensus.cc:348] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:16.707151 1352 raft_consensus.cc:729] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: LEARNER
I20240502 08:18:16.707407 1352 consensus_queue.cc:260] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:16.708050 1352 ts_tablet_manager.cc:1398] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:16.710273 27671 tablet_copy_service.cc:342] P bf8ac71b15f740a0a928911360773442: Request end of tablet copy session 84cda20738d2452da166e31fc6da57a6-33aee3df5eb242ef9cc056d25f6d6a82 received from {username='slave'} at 127.20.207.6:47641
I20240502 08:18:16.710413 27671 tablet_copy_service.cc:433] P bf8ac71b15f740a0a928911360773442: ending tablet copy session 84cda20738d2452da166e31fc6da57a6-33aee3df5eb242ef9cc056d25f6d6a82 on tablet 33aee3df5eb242ef9cc056d25f6d6a82 with peer 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:16.711839 837 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:16.712430 30288 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:16.714761 27109 catalog_manager.cc:5484] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:16.718917 27390 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:16.724031 27648 raft_consensus.cc:2932] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:16.730996 27116 catalog_manager.cc:5484] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 7 to 8, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:16.751794 27115 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:16.752063 27115 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:16.753896 1387 ts_tablet_manager.cc:902] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59: Initiating tablet copy from peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:16.756633 27115 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 85ea6b6d063b40f4b7e5192653cbff09
I20240502 08:18:16.758283 30007 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ea6b6d063b40f4b7e5192653cbff09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:16 UTC) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:16.759017 30400 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ea6b6d063b40f4b7e5192653cbff09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:16 UTC) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:16.759284 1291 tablet_replica.cc:330] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:16.759477 1291 raft_consensus.cc:2229] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:16.759465 27368 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ea6b6d063b40f4b7e5192653cbff09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:16 UTC) from {username='slave'} at 127.0.0.1:40868
I20240502 08:18:16.759660 1291 raft_consensus.cc:2258] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:16.760494 1291 ts_tablet_manager.cc:1875] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:16.770471 1291 ts_tablet_manager.cc:1888] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:16.770658 1291 log.cc:1196] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/85ea6b6d063b40f4b7e5192653cbff09
I20240502 08:18:16.771837 1291 ts_tablet_manager.cc:1909] T 85ea6b6d063b40f4b7e5192653cbff09 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting consensus metadata
I20240502 08:18:16.776326 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet 85ea6b6d063b40f4b7e5192653cbff09 (table rebalancer_dynamic_table_new_name [id=d50d09015f344dc0bf6552a341aaed15]) successfully deleted
I20240502 08:18:16.777415 27255 raft_consensus.cc:2932] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:16.782922 1387 tablet_copy_client.cc:308] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.2:46291
I20240502 08:18:16.783972 27410 tablet_copy_service.cc:140] P 83e171a5e16c4008923f1d43fc5ce7b1: Received BeginTabletCopySession request for tablet 8d309785afd8467dab3418bb63c95030 from peer b01e7c8a302d41078fa39cfef665ff59 ({username='slave'} at 127.20.207.3:46497)
I20240502 08:18:16.784162 27410 tablet_copy_service.cc:161] P 83e171a5e16c4008923f1d43fc5ce7b1: Beginning new tablet copy session on tablet 8d309785afd8467dab3418bb63c95030 from peer b01e7c8a302d41078fa39cfef665ff59 at {username='slave'} at 127.20.207.3:46497: session id = b01e7c8a302d41078fa39cfef665ff59-8d309785afd8467dab3418bb63c95030
I20240502 08:18:16.787577 27410 tablet_copy_source_session.cc:205] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:16.789144 1387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d309785afd8467dab3418bb63c95030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.796622 1271 raft_consensus.cc:1050] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4: attempting to promote NON_VOTER b01e7c8a302d41078fa39cfef665ff59 to VOTER
I20240502 08:18:16.797636 1271 consensus_queue.cc:237] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:16.810515 1387 tablet_copy_client.cc:783] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:16.810890 1387 tablet_copy_client.cc:647] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:16.812667 1387 tablet_copy_client.cc:523] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:16.817858 27518 raft_consensus.cc:1261] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:16.818730 623 consensus_queue.cc:1035] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:16.820936 30025 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:16.828274 27255 consensus_queue.cc:237] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:16.828474 27388 raft_consensus.cc:1261] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:16.829069 623 consensus_queue.cc:1035] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:16.837195 30288 raft_consensus.cc:1261] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:16.837641 1378 consensus_queue.cc:1035] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:16.840767 30025 raft_consensus.cc:1261] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:16.841050 1387 tablet_bootstrap.cc:492] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:16.841486 1378 consensus_queue.cc:1035] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:16.841913 1378 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:16.842286 30159 raft_consensus.cc:1261] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:16.843443 623 consensus_queue.cc:1035] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:16.845911 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2dfa509d47e54b0f9ccacc9954981fb6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:16.846901 30423 raft_consensus.cc:1203] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 08:18:16.846894 30025 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:16.851367 30288 raft_consensus.cc:2932] T 2dfa509d47e54b0f9ccacc9954981fb6 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:16.853720 27116 catalog_manager.cc:5484] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 8 to 9, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:16.856355 1389 tablet_replica.cc:330] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:16.857024 1389 raft_consensus.cc:2229] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:16.858786 1389 raft_consensus.cc:2258] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:16.861590 1389 ts_tablet_manager.cc:1875] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:16.861838 27115 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:16.868132 1390 tablet_replica.cc:330] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:18:16.868378 1390 raft_consensus.cc:2229] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:16.868244 623 raft_consensus.cc:2932] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:16.868623 1390 raft_consensus.cc:2258] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:16.869967 1390 ts_tablet_manager.cc:1875] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:16.872741 27518 raft_consensus.cc:2932] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:16.872864 27390 raft_consensus.cc:2932] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:16.874820 27500 tablet_service.cc:1467] Processing CreateTablet for tablet b1190c1f2e464f229d6b2abd483e6c6a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3618c169ab4a4993bda8a107ae6ebf19]), partition=
I20240502 08:18:16.874955 27629 tablet_service.cc:1467] Processing CreateTablet for tablet b1190c1f2e464f229d6b2abd483e6c6a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3618c169ab4a4993bda8a107ae6ebf19]), partition=
I20240502 08:18:16.875138 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1190c1f2e464f229d6b2abd483e6c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.875281 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1190c1f2e464f229d6b2abd483e6c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.878880 27115 catalog_manager.cc:5484] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:16.879225 27368 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dfa509d47e54b0f9ccacc9954981fb6 with delete_type TABLET_DATA_TOMBSTONED (TS 83e171a5e16c4008923f1d43fc5ce7b1 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:40868
I20240502 08:18:16.882061 30007 tablet_service.cc:1467] Processing CreateTablet for tablet b1190c1f2e464f229d6b2abd483e6c6a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3618c169ab4a4993bda8a107ae6ebf19]), partition=
I20240502 08:18:16.882395 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1190c1f2e464f229d6b2abd483e6c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.888942 30288 consensus_queue.cc:237] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:16.892679 27388 raft_consensus.cc:1261] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:16.893146 1271 consensus_queue.cc:1035] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:16.895051 1270 raft_consensus.cc:2932] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:16.898808 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9bd5657c5cfd4ca4a78cdfdc503b91a8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:16.899886 27390 raft_consensus.cc:2932] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:16.900609 27519 raft_consensus.cc:1261] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:16.901332 1270 consensus_queue.cc:1035] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:16.902036 1389 ts_tablet_manager.cc:1888] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:16.904168 1389 log.cc:1196] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/85ea6b6d063b40f4b7e5192653cbff09
I20240502 08:18:16.905426 1389 ts_tablet_manager.cc:1909] T 85ea6b6d063b40f4b7e5192653cbff09 P 1f0bf4bc3c5146539146b825b048bff3: Deleting consensus metadata
I20240502 08:18:16.908792 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet 85ea6b6d063b40f4b7e5192653cbff09 (table rebalancer_dynamic_table_new_name [id=d50d09015f344dc0bf6552a341aaed15]) successfully deleted
I20240502 08:18:16.909722 1401 tablet_bootstrap.cc:492] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:16.909030 27113 catalog_manager.cc:5484] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:16.914199 30138 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bd5657c5cfd4ca4a78cdfdc503b91a8 with delete_type TABLET_DATA_TOMBSTONED (TS 84cda20738d2452da166e31fc6da57a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:16.915668 1401 tablet_bootstrap.cc:654] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.920903 1399 tablet_bootstrap.cc:492] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:16.926848 27519 raft_consensus.cc:2932] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:16.929752 1390 ts_tablet_manager.cc:1888] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:16.930037 1399 tablet_bootstrap.cc:654] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.931319 1390 log.cc:1196] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/85ea6b6d063b40f4b7e5192653cbff09
I20240502 08:18:16.932344 1390 ts_tablet_manager.cc:1909] T 85ea6b6d063b40f4b7e5192653cbff09 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting consensus metadata
I20240502 08:18:16.933521 1404 tablet_replica.cc:330] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:16.933629 1399 tablet_bootstrap.cc:492] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:16.933773 1399 ts_tablet_manager.cc:1367] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20240502 08:18:16.934372 1399 raft_consensus.cc:348] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:16.920280 1387 tablet_bootstrap.cc:492] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:16.935736 1387 tablet_bootstrap.cc:492] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap complete.
I20240502 08:18:16.935796 1390 tablet_replica.cc:330] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:18:16.935825 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 85ea6b6d063b40f4b7e5192653cbff09 (table rebalancer_dynamic_table_new_name [id=d50d09015f344dc0bf6552a341aaed15]) successfully deleted
I20240502 08:18:16.935992 1387 ts_tablet_manager.cc:1367] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.095s user 0.006s sys 0.003s
I20240502 08:18:16.937127 1390 raft_consensus.cc:2229] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:16.937326 1390 raft_consensus.cc:2258] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:16.937841 1399 raft_consensus.cc:374] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.937927 1399 raft_consensus.cc:729] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.938233 1399 consensus_queue.cc:260] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:16.938740 1399 ts_tablet_manager.cc:1398] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:16.935669 1407 tablet_bootstrap.cc:492] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:16.940464 1390 ts_tablet_manager.cc:1875] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:16.940908 1404 raft_consensus.cc:2229] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:16.941172 1404 pending_rounds.cc:62] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6: Trying to abort 1 pending ops.
I20240502 08:18:16.941788 1387 raft_consensus.cc:348] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:16.941293 1404 pending_rounds.cc:68] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023156825280274432 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "9bd5657c5cfd4ca4a78cdfdc503b91a8" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } } }
I20240502 08:18:16.942960 1404 raft_consensus.cc:2894] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240502 08:18:16.943109 1404 raft_consensus.cc:2866] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 08:18:16.943229 1404 raft_consensus.cc:2258] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:16.943627 1407 tablet_bootstrap.cc:654] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.944391 1404 ts_tablet_manager.cc:1875] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:16.944540 1387 raft_consensus.cc:729] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: LEARNER
I20240502 08:18:16.945315 1387 consensus_queue.cc:260] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:16.951406 1401 tablet_bootstrap.cc:492] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:16.951854 1401 ts_tablet_manager.cc:1367] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20240502 08:18:16.952713 1401 raft_consensus.cc:348] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:16.953027 1401 raft_consensus.cc:374] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.953130 1401 raft_consensus.cc:729] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.953414 1401 consensus_queue.cc:260] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:16.954100 1401 ts_tablet_manager.cc:1398] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:16.955924 1387 ts_tablet_manager.cc:1398] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240502 08:18:16.957304 27410 tablet_copy_service.cc:342] P 83e171a5e16c4008923f1d43fc5ce7b1: Request end of tablet copy session b01e7c8a302d41078fa39cfef665ff59-8d309785afd8467dab3418bb63c95030 received from {username='slave'} at 127.20.207.3:46497
I20240502 08:18:16.957409 27410 tablet_copy_service.cc:433] P 83e171a5e16c4008923f1d43fc5ce7b1: ending tablet copy session b01e7c8a302d41078fa39cfef665ff59-8d309785afd8467dab3418bb63c95030 on tablet 8d309785afd8467dab3418bb63c95030 with peer b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:16.970805 1390 ts_tablet_manager.cc:1888] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:18:16.971014 1390 log.cc:1196] T 2dfa509d47e54b0f9ccacc9954981fb6 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/2dfa509d47e54b0f9ccacc9954981fb6
I20240502 08:18:16.973591 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 2dfa509d47e54b0f9ccacc9954981fb6 (table rebalance_test_table_19 [id=7f9af7f2cb49474eb721698045f06a63]) successfully deleted
I20240502 08:18:16.974664 1404 ts_tablet_manager.cc:1888] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:16.974876 1404 log.cc:1196] T 9bd5657c5cfd4ca4a78cdfdc503b91a8 P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/9bd5657c5cfd4ca4a78cdfdc503b91a8
I20240502 08:18:16.975787 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet 9bd5657c5cfd4ca4a78cdfdc503b91a8 (table rebalancer_extra_table_28 [id=bb73dde083304fce8a7da7ac8fdd1d8a]) successfully deleted
I20240502 08:18:16.988885 1413 ts_tablet_manager.cc:902] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:16.999856 1407 tablet_bootstrap.cc:492] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:17.000021 1407 ts_tablet_manager.cc:1367] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.006s
I20240502 08:18:17.003850 1407 raft_consensus.cc:348] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:17.004673 1407 raft_consensus.cc:374] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.004808 1407 raft_consensus.cc:729] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.005100 1407 consensus_queue.cc:260] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:17.008652 1407 ts_tablet_manager.cc:1398] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 08:18:17.017030 1413 tablet_copy_client.cc:308] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:18:17.017612 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet a793eaaeba89458baf8af7caf3f24c70 from peer 49c5c8f70e494edd9fcda40cc0af6cc4 ({username='slave'} at 127.20.207.7:45231)
I20240502 08:18:17.017737 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet a793eaaeba89458baf8af7caf3f24c70 from peer 49c5c8f70e494edd9fcda40cc0af6cc4 at {username='slave'} at 127.20.207.7:45231: session id = 49c5c8f70e494edd9fcda40cc0af6cc4-a793eaaeba89458baf8af7caf3f24c70
I20240502 08:18:17.019707 30157 raft_consensus.cc:1203] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:18:17.020473 27278 tablet_copy_source_session.cc:205] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:17.028573 1413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a793eaaeba89458baf8af7caf3f24c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.047683 1413 tablet_copy_client.cc:783] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:17.048503 1413 tablet_copy_client.cc:647] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:17.052100 27109 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:17.056550 1413 tablet_copy_client.cc:523] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:17.060379 932 raft_consensus.cc:482] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:17.060534 932 raft_consensus.cc:504] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:17.061336 932 leader_election.cc:288] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:17.062250 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1190c1f2e464f229d6b2abd483e6c6a" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:17.062443 27650 raft_consensus.cc:2454] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:18:17.063359 27456 leader_election.cc:302] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:17.063881 1185 raft_consensus.cc:2781] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:17.063987 1185 raft_consensus.cc:482] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:17.064055 1185 raft_consensus.cc:3037] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.066200 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1190c1f2e464f229d6b2abd483e6c6a" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:17.066417 30025 raft_consensus.cc:2454] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:18:17.070201 30007 tablet_service.cc:1467] Processing CreateTablet for tablet c3a55662c6e346f0b73df334be4b14e3 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=4aae824440954ca3be57248f5f1f6249]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:17.070348 27629 tablet_service.cc:1467] Processing CreateTablet for tablet 119ad5932c314235b85177cceeb2277c (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=4aae824440954ca3be57248f5f1f6249]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:17.070659 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a55662c6e346f0b73df334be4b14e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.070722 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 119ad5932c314235b85177cceeb2277c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.074366 27368 tablet_service.cc:1467] Processing CreateTablet for tablet 119ad5932c314235b85177cceeb2277c (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=4aae824440954ca3be57248f5f1f6249]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:17.074481 27370 tablet_service.cc:1467] Processing CreateTablet for tablet c3a55662c6e346f0b73df334be4b14e3 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=4aae824440954ca3be57248f5f1f6249]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:17.074746 27368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 119ad5932c314235b85177cceeb2277c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.076766 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a55662c6e346f0b73df334be4b14e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.079484 1185 raft_consensus.cc:504] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:17.080194 1185 leader_election.cc:288] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:17.080579 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1190c1f2e464f229d6b2abd483e6c6a" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:17.080770 27650 raft_consensus.cc:3037] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.083218 27650 raft_consensus.cc:2454] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:18:17.083592 27366 tablet_service.cc:1467] Processing CreateTablet for tablet a52fe5caeb364b868a1965633a7da09e (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=4aae824440954ca3be57248f5f1f6249]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:17.083969 27366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52fe5caeb364b868a1965633a7da09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.084194 30138 tablet_service.cc:1467] Processing CreateTablet for tablet a52fe5caeb364b868a1965633a7da09e (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=4aae824440954ca3be57248f5f1f6249]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:17.084518 27456 leader_election.cc:302] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:17.084472 30400 tablet_service.cc:1467] Processing CreateTablet for tablet a52fe5caeb364b868a1965633a7da09e (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=4aae824440954ca3be57248f5f1f6249]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:17.084628 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52fe5caeb364b868a1965633a7da09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.084859 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52fe5caeb364b868a1965633a7da09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.090178 932 raft_consensus.cc:2781] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:17.090302 932 raft_consensus.cc:686] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:18:17.090550 932 consensus_queue.cc:237] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:17.093425 1407 tablet_bootstrap.cc:492] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:17.095453 1407 tablet_bootstrap.cc:654] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.100039 1429 tablet_bootstrap.cc:492] T a52fe5caeb364b868a1965633a7da09e P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:17.105782 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 119ad5932c314235b85177cceeb2277c (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=4aae824440954ca3be57248f5f1f6249]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:17.106233 30270 tablet_service.cc:1467] Processing CreateTablet for tablet c3a55662c6e346f0b73df334be4b14e3 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=4aae824440954ca3be57248f5f1f6249]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:17.106434 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 119ad5932c314235b85177cceeb2277c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.106681 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a55662c6e346f0b73df334be4b14e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.116799 1429 tablet_bootstrap.cc:654] T a52fe5caeb364b868a1965633a7da09e P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.124331 1399 tablet_bootstrap.cc:492] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:17.125869 1399 tablet_bootstrap.cc:654] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.126848 1435 tablet_bootstrap.cc:492] T 119ad5932c314235b85177cceeb2277c P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:17.128302 1435 tablet_bootstrap.cc:654] T 119ad5932c314235b85177cceeb2277c P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.128858 1429 tablet_bootstrap.cc:492] T a52fe5caeb364b868a1965633a7da09e P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:18:17.129155 1431 tablet_bootstrap.cc:492] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:17.129302 1429 ts_tablet_manager.cc:1367] T a52fe5caeb364b868a1965633a7da09e P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20240502 08:18:17.130826 1429 raft_consensus.cc:348] T a52fe5caeb364b868a1965633a7da09e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:17.131024 1429 raft_consensus.cc:374] T a52fe5caeb364b868a1965633a7da09e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.131074 1429 raft_consensus.cc:729] T a52fe5caeb364b868a1965633a7da09e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.131479 1429 consensus_queue.cc:260] T a52fe5caeb364b868a1965633a7da09e P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:17.132086 1429 ts_tablet_manager.cc:1398] T a52fe5caeb364b868a1965633a7da09e P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:17.133148 1429 tablet_bootstrap.cc:492] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:17.134371 1429 tablet_bootstrap.cc:654] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.138167 1431 tablet_bootstrap.cc:654] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.138381 1413 tablet_bootstrap.cc:492] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:17.140949 1431 tablet_bootstrap.cc:492] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:17.141105 1431 ts_tablet_manager.cc:1367] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.002s
I20240502 08:18:17.142272 1431 raft_consensus.cc:348] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:17.142426 1431 raft_consensus.cc:374] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.142886 1429 tablet_bootstrap.cc:492] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:18:17.142993 1429 ts_tablet_manager.cc:1367] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240502 08:18:17.143489 1429 raft_consensus.cc:348] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:17.143618 1429 raft_consensus.cc:374] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.143662 1429 raft_consensus.cc:729] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.143838 1431 raft_consensus.cc:729] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.144129 1431 consensus_queue.cc:260] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:17.144661 1431 ts_tablet_manager.cc:1398] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:17.145773 1429 consensus_queue.cc:260] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:17.147778 1429 ts_tablet_manager.cc:1398] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:18:17.148571 1399 tablet_bootstrap.cc:492] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:17.148795 1399 ts_tablet_manager.cc:1367] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.002s
I20240502 08:18:17.149497 1399 raft_consensus.cc:348] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:17.149760 1399 raft_consensus.cc:374] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.149878 1399 raft_consensus.cc:729] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.150210 1399 consensus_queue.cc:260] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:17.150759 1399 ts_tablet_manager.cc:1398] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:17.151412 1429 tablet_bootstrap.cc:492] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:17.153134 1429 tablet_bootstrap.cc:654] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.167832 1429 tablet_bootstrap.cc:492] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:18:17.167976 1429 ts_tablet_manager.cc:1367] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20240502 08:18:17.168876 1429 raft_consensus.cc:348] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:17.169147 1429 raft_consensus.cc:374] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.169217 1429 raft_consensus.cc:729] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.171777 27109 catalog_manager.cc:5484] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:17.174453 1407 tablet_bootstrap.cc:492] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:17.174683 1407 ts_tablet_manager.cc:1367] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.081s user 0.006s sys 0.006s
I20240502 08:18:17.175335 1429 consensus_queue.cc:260] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:17.176219 1407 raft_consensus.cc:348] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:17.176470 1407 raft_consensus.cc:374] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.176548 1407 raft_consensus.cc:729] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.178035 1407 consensus_queue.cc:260] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:17.178339 1429 ts_tablet_manager.cc:1398] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20240502 08:18:17.178689 1407 ts_tablet_manager.cc:1398] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:17.184990 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1190c1f2e464f229d6b2abd483e6c6a" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:17.185194 30025 raft_consensus.cc:3037] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.187007 27519 raft_consensus.cc:1203] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:17.188091 30025 raft_consensus.cc:2454] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:18:17.197700 1432 tablet_bootstrap.cc:492] T a52fe5caeb364b868a1965633a7da09e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:17.199772 1432 tablet_bootstrap.cc:654] T a52fe5caeb364b868a1965633a7da09e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.211488 1432 tablet_bootstrap.cc:492] T a52fe5caeb364b868a1965633a7da09e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:17.211674 1432 ts_tablet_manager.cc:1367] T a52fe5caeb364b868a1965633a7da09e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20240502 08:18:17.212760 1432 raft_consensus.cc:348] T a52fe5caeb364b868a1965633a7da09e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:17.213032 1432 raft_consensus.cc:374] T a52fe5caeb364b868a1965633a7da09e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.213153 1432 raft_consensus.cc:729] T a52fe5caeb364b868a1965633a7da09e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.213375 1435 tablet_bootstrap.cc:492] T 119ad5932c314235b85177cceeb2277c P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:17.213416 1432 consensus_queue.cc:260] T a52fe5caeb364b868a1965633a7da09e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:17.213610 1435 ts_tablet_manager.cc:1367] T 119ad5932c314235b85177cceeb2277c P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.087s user 0.005s sys 0.012s
I20240502 08:18:17.215294 1435 raft_consensus.cc:348] T 119ad5932c314235b85177cceeb2277c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:17.215646 1435 raft_consensus.cc:374] T 119ad5932c314235b85177cceeb2277c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.215824 1435 raft_consensus.cc:729] T 119ad5932c314235b85177cceeb2277c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.216287 1435 consensus_queue.cc:260] T 119ad5932c314235b85177cceeb2277c P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:17.220628 1432 ts_tablet_manager.cc:1398] T a52fe5caeb364b868a1965633a7da09e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 08:18:17.236043 1435 ts_tablet_manager.cc:1398] T 119ad5932c314235b85177cceeb2277c P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240502 08:18:17.236420 1435 tablet_bootstrap.cc:492] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:17.249991 1435 tablet_bootstrap.cc:654] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.271149 1435 tablet_bootstrap.cc:492] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:17.271466 1435 ts_tablet_manager.cc:1367] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.001s
I20240502 08:18:17.272549 1435 raft_consensus.cc:348] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:17.273104 1435 raft_consensus.cc:374] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.273252 1435 raft_consensus.cc:729] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.273669 1435 consensus_queue.cc:260] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:17.276249 1435 ts_tablet_manager.cc:1398] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:18:17.291232 1413 tablet_bootstrap.cc:492] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:17.291692 1413 tablet_bootstrap.cc:492] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap complete.
I20240502 08:18:17.291954 1413 ts_tablet_manager.cc:1367] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.154s user 0.045s sys 0.009s
I20240502 08:18:17.292802 1413 raft_consensus.cc:348] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:17.293056 1413 raft_consensus.cc:729] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: LEARNER
I20240502 08:18:17.293277 1413 consensus_queue.cc:260] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:17.294530 1413 ts_tablet_manager.cc:1398] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:17.295292 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-a793eaaeba89458baf8af7caf3f24c70 received from {username='slave'} at 127.20.207.7:45231
I20240502 08:18:17.295403 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-a793eaaeba89458baf8af7caf3f24c70 on tablet a793eaaeba89458baf8af7caf3f24c70 with peer 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:17.297802 27116 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:17.300524 1300 raft_consensus.cc:482] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:17.300673 1300 raft_consensus.cc:504] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:17.301453 1300 leader_election.cc:288] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:17.302156 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52fe5caeb364b868a1965633a7da09e" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:18:17.302394 27388 raft_consensus.cc:2454] T a52fe5caeb364b868a1965633a7da09e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:17.305630 30095 leader_election.cc:302] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:17.305881 1300 raft_consensus.cc:2781] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:17.305985 1300 raft_consensus.cc:482] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:17.306110 1300 raft_consensus.cc:3037] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.307787 1294 raft_consensus.cc:482] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:17.307910 1294 raft_consensus.cc:504] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:17.308607 1294 leader_election.cc:288] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:17.308617 1300 raft_consensus.cc:504] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:17.309324 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ad5932c314235b85177cceeb2277c" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:17.309549 1300 leader_election.cc:288] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:17.309535 30288 raft_consensus.cc:2454] T 119ad5932c314235b85177cceeb2277c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:18:17.309937 27323 leader_election.cc:302] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:17.310140 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ad5932c314235b85177cceeb2277c" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:17.310237 1294 raft_consensus.cc:2781] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:17.310297 1294 raft_consensus.cc:482] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:17.310355 1294 raft_consensus.cc:3037] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.310346 27650 raft_consensus.cc:2454] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:18:17.310961 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52fe5caeb364b868a1965633a7da09e" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:17.311195 30423 raft_consensus.cc:2454] T a52fe5caeb364b868a1965633a7da09e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:17.312775 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52fe5caeb364b868a1965633a7da09e" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:18:17.312839 1294 raft_consensus.cc:504] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:17.312929 27388 raft_consensus.cc:3037] T a52fe5caeb364b868a1965633a7da09e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.313364 1294 leader_election.cc:288] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:17.314419 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ad5932c314235b85177cceeb2277c" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:17.314589 30288 raft_consensus.cc:3037] T 119ad5932c314235b85177cceeb2277c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.315476 27388 raft_consensus.cc:2454] T a52fe5caeb364b868a1965633a7da09e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:17.316018 30095 leader_election.cc:302] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:17.317147 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119ad5932c314235b85177cceeb2277c" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:17.317312 27650 raft_consensus.cc:3037] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.318127 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52fe5caeb364b868a1965633a7da09e" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:17.318311 30423 raft_consensus.cc:3037] T a52fe5caeb364b868a1965633a7da09e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.321020 30423 raft_consensus.cc:2454] T a52fe5caeb364b868a1965633a7da09e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:17.321755 30288 raft_consensus.cc:2454] T 119ad5932c314235b85177cceeb2277c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:18:17.322168 27323 leader_election.cc:302] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:17.322441 1294 raft_consensus.cc:2781] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:17.322537 1294 raft_consensus.cc:686] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:18:17.322795 1294 consensus_queue.cc:237] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:17.323696 27650 raft_consensus.cc:2454] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:18:17.329871 300 raft_consensus.cc:2781] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:17.330053 300 raft_consensus.cc:686] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:17.330469 300 consensus_queue.cc:237] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } }
I20240502 08:18:17.333114 30025 raft_consensus.cc:1261] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:17.334070 932 consensus_queue.cc:1035] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.340000 27650 raft_consensus.cc:1261] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:17.340801 932 consensus_queue.cc:1035] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.348095 27109 catalog_manager.cc:5484] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:17.392699 1468 tablet.cc:1722] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:17.393332 1468 tablet.cc:1620] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59: MemRowSet was empty: no flush needed.
I20240502 08:18:17.393702 27109 catalog_manager.cc:5484] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:17.401320 27113 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet b1190c1f2e464f229d6b2abd483e6c6a (table rebalancer_dynamic_table [id=3618c169ab4a4993bda8a107ae6ebf19]). Expected version 1 got 0
I20240502 08:18:17.414316 1469 tablet.cc:1722] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:17.414961 1469 tablet.cc:1620] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:17.415413 1378 raft_consensus.cc:1050] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad to VOTER
I20240502 08:18:17.416236 1378 consensus_queue.cc:237] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:17.420231 30288 raft_consensus.cc:1203] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:18:17.426311 27116 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=3618c169ab4a4993bda8a107ae6ebf19] alter complete (version 1)
I20240502 08:18:17.430421 1026 raft_consensus.cc:482] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:17.430573 1026 raft_consensus.cc:504] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:17.431243 1026 leader_election.cc:288] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:17.431941 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a55662c6e346f0b73df334be4b14e3" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:18:17.432174 27388 raft_consensus.cc:2454] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:17.433287 29963 leader_election.cc:302] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:17.440255 1129 raft_consensus.cc:2781] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:17.440433 1129 raft_consensus.cc:482] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:17.440554 1129 raft_consensus.cc:3037] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.443228 1129 raft_consensus.cc:504] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:17.444463 1271 raft_consensus.cc:482] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:17.444649 1271 raft_consensus.cc:504] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:17.445305 1271 leader_election.cc:288] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:17.445745 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a55662c6e346f0b73df334be4b14e3" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:17.446142 30025 raft_consensus.cc:2379] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in current term 1: Already voted for candidate 1f0bf4bc3c5146539146b825b048bff3 in this term.
I20240502 08:18:17.447432 1129 leader_election.cc:288] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:17.448015 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a55662c6e346f0b73df334be4b14e3" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:17.448204 30289 raft_consensus.cc:2454] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:17.450464 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a55662c6e346f0b73df334be4b14e3" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:18:17.450673 27389 raft_consensus.cc:2454] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:17.456466 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a55662c6e346f0b73df334be4b14e3" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:18:17.456619 27388 raft_consensus.cc:3037] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.459460 27388 raft_consensus.cc:2454] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:17.460011 29963 leader_election.cc:302] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:17.460449 1474 raft_consensus.cc:2781] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:17.460608 1474 raft_consensus.cc:686] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:18:17.462579 1474 consensus_queue.cc:237] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:17.465832 30225 leader_election.cc:302] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1; no voters: 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:18:17.466636 623 raft_consensus.cc:2781] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:17.466724 623 raft_consensus.cc:482] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:17.466807 623 raft_consensus.cc:3037] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.473923 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a55662c6e346f0b73df334be4b14e3" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:17.481914 623 raft_consensus.cc:504] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:17.482780 623 leader_election.cc:288] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:17.483609 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a55662c6e346f0b73df334be4b14e3" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:17.484370 30288 raft_consensus.cc:2379] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1f0bf4bc3c5146539146b825b048bff3 in current term 1: Already voted for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in this term.
I20240502 08:18:17.484628 27115 catalog_manager.cc:5484] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:17.486321 27389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a55662c6e346f0b73df334be4b14e3" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:18:17.486533 27389 raft_consensus.cc:2379] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in current term 1: Already voted for candidate 1f0bf4bc3c5146539146b825b048bff3 in this term.
I20240502 08:18:17.487361 30225 leader_election.cc:302] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4; no voters: 1f0bf4bc3c5146539146b825b048bff3, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:17.488673 1271 raft_consensus.cc:2726] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:18:17.491675 30423 raft_consensus.cc:1261] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:17.493649 32172 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:17.495922 27115 catalog_manager.cc:5576] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309) does not have the latest schema for tablet b1190c1f2e464f229d6b2abd483e6c6a (table rebalancer_dynamic_table [id=3618c169ab4a4993bda8a107ae6ebf19]). Expected version 1 got 0
I20240502 08:18:17.503979 1480 tablet.cc:1722] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:17.506973 27115 catalog_manager.cc:5576] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309) does not have the latest schema for tablet b1190c1f2e464f229d6b2abd483e6c6a (table rebalancer_dynamic_table [id=3618c169ab4a4993bda8a107ae6ebf19]). Expected version 1 got 0
I20240502 08:18:17.507963 30288 raft_consensus.cc:1261] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:17.508962 1480 tablet.cc:1620] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442: MemRowSet was empty: no flush needed.
I20240502 08:18:17.510460 32172 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:17.519681 27648 raft_consensus.cc:1261] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:17.520413 32172 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:17.540302 1378 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:17.541939 30423 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:17.542344 30288 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:17.544915 27650 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:17.548604 27110 catalog_manager.cc:5484] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 8 to 9, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:17.562062 27255 consensus_queue.cc:237] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:17.564401 30288 raft_consensus.cc:1261] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:17.565168 837 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:17.565454 30423 raft_consensus.cc:1261] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:17.566359 837 consensus_queue.cc:1035] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:17.568580 1378 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:17.569802 30288 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:17.575543 30423 raft_consensus.cc:2932] T 903502e57ab2430383807b0d46a39847 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:17.576385 27116 catalog_manager.cc:5484] T 903502e57ab2430383807b0d46a39847 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 9 to 10, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New cstate: current_term: 2 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:17.576923 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 903502e57ab2430383807b0d46a39847 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:17.579558 27115 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:17.586675 27629 tablet_service.cc:1514] Processing DeleteTablet for tablet 903502e57ab2430383807b0d46a39847 with delete_type TABLET_DATA_TOMBSTONED (TS bf8ac71b15f740a0a928911360773442 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44788
I20240502 08:18:17.588263 1495 tablet_replica.cc:330] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:18:17.588485 1495 raft_consensus.cc:2229] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:17.588822 1495 raft_consensus.cc:2258] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:17.590296 1495 ts_tablet_manager.cc:1875] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:17.593158 1468 tablet.cc:1722] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:17.593627 1468 tablet.cc:1620] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59: MemRowSet was empty: no flush needed.
I20240502 08:18:17.593835 1480 tablet.cc:1722] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:17.594272 1480 tablet.cc:1620] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442: MemRowSet was empty: no flush needed.
I20240502 08:18:17.596036 1469 tablet.cc:1722] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:17.596447 1469 tablet.cc:1620] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:17.604367 1495 ts_tablet_manager.cc:1888] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 08:18:17.604647 27115 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=3618c169ab4a4993bda8a107ae6ebf19] alter complete (version 2)
I20240502 08:18:17.608989 1495 log.cc:1196] T 903502e57ab2430383807b0d46a39847 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/903502e57ab2430383807b0d46a39847
I20240502 08:18:17.609884 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 903502e57ab2430383807b0d46a39847 (table rebalance_test_table_13 [id=28a67fad5bff471981e8cdf2b2a2b34a]) successfully deleted
I20240502 08:18:17.618443 1314 raft_consensus.cc:1050] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442: attempting to promote NON_VOTER 84cda20738d2452da166e31fc6da57a6 to VOTER
I20240502 08:18:17.619230 1314 consensus_queue.cc:237] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:17.622659 30157 raft_consensus.cc:1261] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:17.622888 30423 raft_consensus.cc:1261] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:17.623139 27389 raft_consensus.cc:1261] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:17.623595 1502 consensus_queue.cc:1035] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:17.623924 1314 consensus_queue.cc:1035] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:17.624225 1311 consensus_queue.cc:1035] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:17.648470 1311 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:17.649919 30156 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:17.653092 30423 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:17.655467 27110 catalog_manager.cc:5484] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:17.657827 27389 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:17.661919 27650 consensus_queue.cc:237] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:17.663992 30157 raft_consensus.cc:1261] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:17.664474 1311 consensus_queue.cc:1035] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:17.668359 30418 raft_consensus.cc:1261] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:17.669059 1314 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:17.672227 1502 consensus_queue.cc:1035] T 33aee3df5eb242ef9cc056d25f6d6a82 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:17.673214 30156 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:17.674196 30423 raft_consensus.cc:2932] T 33aee3df5eb242ef9cc056d25f6d6a82 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:17.682744 27116 catalog_manager.cc:5484] T 33aee3df5eb242ef9cc056d25f6d6a82 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 8 to 9, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:17.688573 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 33aee3df5eb242ef9cc056d25f6d6a82 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:17.696502 27370 tablet_service.cc:1514] Processing DeleteTablet for tablet 33aee3df5eb242ef9cc056d25f6d6a82 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 8 (current committed config index is 9)) from {username='slave'} at 127.0.0.1:40868
I20240502 08:18:17.708379 1519 tablet_replica.cc:330] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:18:17.708631 1519 raft_consensus.cc:2229] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:17.708878 1519 raft_consensus.cc:2258] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:17.709645 27116 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:17.709789 1519 ts_tablet_manager.cc:1875] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:17.716571 1519 ts_tablet_manager.cc:1888] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:18:17.717156 1519 log.cc:1196] T 33aee3df5eb242ef9cc056d25f6d6a82 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/33aee3df5eb242ef9cc056d25f6d6a82
I20240502 08:18:17.717937 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 33aee3df5eb242ef9cc056d25f6d6a82 (table rebalance_test_table_9 [id=a65036830fa64a22b9f4efd6d5bd4f9f]) successfully deleted
I20240502 08:18:17.720059 1468 tablet.cc:1722] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:17.720440 1469 tablet.cc:1722] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:17.721158 1480 tablet.cc:1722] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:17.731268 27109 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=3618c169ab4a4993bda8a107ae6ebf19] alter complete (version 3)
I20240502 08:18:17.738222 30423 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fefcc470084f4ec9be6ba45d3ba22ce5"
dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58696
I20240502 08:18:17.738502 30423 raft_consensus.cc:595] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Received request to transfer leadership
I20240502 08:18:17.738941 1294 consensus_queue.cc:1035] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.748786 1400 raft_consensus.cc:979] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: : Instructing follower b01e7c8a302d41078fa39cfef665ff59 to start an election
I20240502 08:18:17.748939 1400 raft_consensus.cc:1067] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Signalling peer b01e7c8a302d41078fa39cfef665ff59 to start an election
I20240502 08:18:17.749797 27519 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fefcc470084f4ec9be6ba45d3ba22ce5"
dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
from {username='slave'} at 127.20.207.8:38115
I20240502 08:18:17.750069 27519 raft_consensus.cc:482] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:17.750182 27519 raft_consensus.cc:3037] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:17.752934 27519 raft_consensus.cc:504] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:17.754519 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefcc470084f4ec9be6ba45d3ba22ce5" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:17.754751 30157 raft_consensus.cc:3037] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:17.757411 30157 raft_consensus.cc:2454] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 2.
I20240502 08:18:17.758440 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefcc470084f4ec9be6ba45d3ba22ce5" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:17.758689 30423 raft_consensus.cc:3032] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:17.758759 30423 raft_consensus.cc:729] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:18:17.759104 30423 consensus_queue.cc:260] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:17.759505 30423 raft_consensus.cc:3037] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:17.760373 1294 consensus_queue.cc:1035] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.762148 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefcc470084f4ec9be6ba45d3ba22ce5" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:17.762337 27650 raft_consensus.cc:3037] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:17.765000 27650 raft_consensus.cc:2454] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 2.
I20240502 08:18:17.768317 27456 leader_election.cc:302] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6, b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:17.768620 1185 raft_consensus.cc:2781] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:17.768764 1185 raft_consensus.cc:686] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [term 2 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:18:17.769064 1185 consensus_queue.cc:237] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:17.771541 30423 raft_consensus.cc:2454] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 2.
I20240502 08:18:17.771755 27519 leader_election.cc:288] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 2 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:17.778465 1294 raft_consensus.cc:1050] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1: attempting to promote NON_VOTER b01e7c8a302d41078fa39cfef665ff59 to VOTER
I20240502 08:18:17.779558 1294 consensus_queue.cc:237] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:17.782925 27519 raft_consensus.cc:1261] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:17.783381 1538 consensus_queue.cc:1035] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:17.788022 30025 raft_consensus.cc:1261] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:17.790058 1294 consensus_queue.cc:1035] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:17.790573 30423 raft_consensus.cc:1261] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:17.791141 1294 consensus_queue.cc:1035] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:17.823855 1294 raft_consensus.cc:2932] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:17.824343 27116 catalog_manager.cc:5484] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 1 to 2, leader changed from 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 2 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:17.831286 30423 raft_consensus.cc:2932] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:17.832962 27519 raft_consensus.cc:2932] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:17.837069 27116 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:17.837250 27116 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:17.841265 27110 catalog_manager.cc:5484] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:17.842478 27116 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet b1190c1f2e464f229d6b2abd483e6c6a
I20240502 08:18:17.843446 27629 tablet_service.cc:1514] Processing DeleteTablet for tablet b1190c1f2e464f229d6b2abd483e6c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:17 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:18:17.843710 1495 tablet_replica.cc:330] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:18:17.843739 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet b1190c1f2e464f229d6b2abd483e6c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:17 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:17.843909 1495 raft_consensus.cc:2229] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:17.844022 30007 tablet_service.cc:1514] Processing DeleteTablet for tablet b1190c1f2e464f229d6b2abd483e6c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:17 UTC) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:17.844076 1495 raft_consensus.cc:2258] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:17.844455 1546 tablet_replica.cc:330] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:17.844678 1546 raft_consensus.cc:2229] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:17.845054 1546 raft_consensus.cc:2258] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:17.845115 1495 ts_tablet_manager.cc:1875] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:17.847218 30025 raft_consensus.cc:2932] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:17.849368 1495 ts_tablet_manager.cc:1888] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:17.849452 1495 log.cc:1196] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/b1190c1f2e464f229d6b2abd483e6c6a
I20240502 08:18:17.849694 1495 ts_tablet_manager.cc:1909] T b1190c1f2e464f229d6b2abd483e6c6a P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:18:17.851469 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet b1190c1f2e464f229d6b2abd483e6c6a (table rebalancer_dynamic_table_new_name [id=3618c169ab4a4993bda8a107ae6ebf19]) successfully deleted
I20240502 08:18:17.851940 27389 consensus_queue.cc:237] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:17.852672 1546 ts_tablet_manager.cc:1875] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:17.853672 30423 raft_consensus.cc:1261] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:17.853634 27519 raft_consensus.cc:1261] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:17.854457 1294 consensus_queue.cc:1035] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:17.855051 1294 consensus_queue.cc:1035] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:17.858386 1539 raft_consensus.cc:2932] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:17.862632 27519 raft_consensus.cc:2932] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:17.864298 30418 raft_consensus.cc:2932] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:17.868386 27110 catalog_manager.cc:5484] T 8d309785afd8467dab3418bb63c95030 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:17.869071 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8d309785afd8467dab3418bb63c95030 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:17.872537 30006 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d309785afd8467dab3418bb63c95030 with delete_type TABLET_DATA_TOMBSTONED (TS 1f0bf4bc3c5146539146b825b048bff3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:17.884182 1546 ts_tablet_manager.cc:1888] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:17.886271 1547 tablet_replica.cc:330] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:17.886785 1547 raft_consensus.cc:2229] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:17.887040 1547 raft_consensus.cc:2258] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:17.888223 1547 ts_tablet_manager.cc:1875] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:17.890125 1546 log.cc:1196] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/b1190c1f2e464f229d6b2abd483e6c6a
I20240502 08:18:17.890554 1546 ts_tablet_manager.cc:1909] T b1190c1f2e464f229d6b2abd483e6c6a P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:18:17.895284 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet b1190c1f2e464f229d6b2abd483e6c6a (table rebalancer_dynamic_table_new_name [id=3618c169ab4a4993bda8a107ae6ebf19]) successfully deleted
I20240502 08:18:17.896941 1547 ts_tablet_manager.cc:1888] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:17.897126 1547 log.cc:1196] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/b1190c1f2e464f229d6b2abd483e6c6a
I20240502 08:18:17.897665 1547 ts_tablet_manager.cc:1909] T b1190c1f2e464f229d6b2abd483e6c6a P 1f0bf4bc3c5146539146b825b048bff3: Deleting consensus metadata
I20240502 08:18:17.898859 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet b1190c1f2e464f229d6b2abd483e6c6a (table rebalancer_dynamic_table_new_name [id=3618c169ab4a4993bda8a107ae6ebf19]) successfully deleted
I20240502 08:18:17.899132 1547 tablet_replica.cc:330] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:17.899446 1547 raft_consensus.cc:2229] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:17.899775 1547 raft_consensus.cc:2258] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:17.901355 1547 ts_tablet_manager.cc:1875] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:17.912341 32172 raft_consensus.cc:1050] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 to VOTER
I20240502 08:18:17.913111 32172 consensus_queue.cc:237] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:17.917964 1547 ts_tablet_manager.cc:1888] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:17.918184 1547 log.cc:1196] T 8d309785afd8467dab3418bb63c95030 P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/8d309785afd8467dab3418bb63c95030
I20240502 08:18:17.920140 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet 8d309785afd8467dab3418bb63c95030 (table rebalancer_extra_table_22 [id=77ecd1036c454af293decc2ea5173f97]) successfully deleted
I20240502 08:18:17.920961 300 consensus_queue.cc:1035] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.921732 30290 raft_consensus.cc:1261] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:17.921983 30157 raft_consensus.cc:1261] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:17.923616 1129 consensus_queue.cc:1035] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.923754 1378 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:17.926792 27388 raft_consensus.cc:1261] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:17.927361 1378 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:17.927788 837 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:17.940606 300 consensus_queue.cc:1035] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.947225 27115 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:17.955879 27500 tablet_service.cc:1467] Processing CreateTablet for tablet b92831cb328345c28d3fec345b9bc1cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c7bc9125709d40d2b0ba8a270d9175c1]), partition=
I20240502 08:18:17.956203 30007 tablet_service.cc:1467] Processing CreateTablet for tablet b92831cb328345c28d3fec345b9bc1cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c7bc9125709d40d2b0ba8a270d9175c1]), partition=
I20240502 08:18:17.956265 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b92831cb328345c28d3fec345b9bc1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.956542 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b92831cb328345c28d3fec345b9bc1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.958070 1129 consensus_queue.cc:1035] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.962476 27629 tablet_service.cc:1467] Processing CreateTablet for tablet b92831cb328345c28d3fec345b9bc1cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c7bc9125709d40d2b0ba8a270d9175c1]), partition=
I20240502 08:18:17.962792 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b92831cb328345c28d3fec345b9bc1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.992659 1562 tablet_bootstrap.cc:492] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:17.995352 1562 tablet_bootstrap.cc:654] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.001550 1563 tablet_bootstrap.cc:492] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:18.003625 1563 tablet_bootstrap.cc:654] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.005914 32172 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:18.007356 27390 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:18.006831 30157 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:18.008654 1563 tablet_bootstrap.cc:492] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:18.008783 1563 ts_tablet_manager.cc:1367] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 08:18:18.012879 30288 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:18.014727 27115 catalog_manager.cc:5484] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 7 to 8, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:18.016456 1564 tablet_bootstrap.cc:492] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:18.020892 1564 tablet_bootstrap.cc:654] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.010030 1563 raft_consensus.cc:348] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.023587 1563 raft_consensus.cc:374] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.023753 1563 raft_consensus.cc:729] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.024106 1563 consensus_queue.cc:260] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.046960 1564 tablet_bootstrap.cc:492] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:18.046994 1562 tablet_bootstrap.cc:492] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:18.047192 1562 ts_tablet_manager.cc:1367] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.000s
I20240502 08:18:18.047209 1564 ts_tablet_manager.cc:1367] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240502 08:18:18.047952 1564 raft_consensus.cc:348] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.048226 1564 raft_consensus.cc:374] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.048131 1562 raft_consensus.cc:348] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.048359 1562 raft_consensus.cc:374] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.048411 1564 raft_consensus.cc:729] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.048462 1562 raft_consensus.cc:729] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.048679 1562 consensus_queue.cc:260] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.048772 1564 consensus_queue.cc:260] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.050132 1563 ts_tablet_manager.cc:1398] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.041s user 0.000s sys 0.002s
I20240502 08:18:18.050940 1562 ts_tablet_manager.cc:1398] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:18.068374 27115 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:18.078619 30400 tablet_service.cc:1467] Processing CreateTablet for tablet d122f68d36024614a017e8cc3a1224c1 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8d8448e41c7842ff8ddf670040a9aebd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.078612 30007 tablet_service.cc:1467] Processing CreateTablet for tablet d122f68d36024614a017e8cc3a1224c1 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8d8448e41c7842ff8ddf670040a9aebd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.078696 27238 tablet_service.cc:1467] Processing CreateTablet for tablet ecd2d7e4899d4755ad575feb60c24d4b (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8d8448e41c7842ff8ddf670040a9aebd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.078740 27500 tablet_service.cc:1467] Processing CreateTablet for tablet d122f68d36024614a017e8cc3a1224c1 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8d8448e41c7842ff8ddf670040a9aebd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.079022 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd2d7e4899d4755ad575feb60c24d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.079037 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d122f68d36024614a017e8cc3a1224c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.079078 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d122f68d36024614a017e8cc3a1224c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.079102 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d122f68d36024614a017e8cc3a1224c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.083353 27629 tablet_service.cc:1467] Processing CreateTablet for tablet ecd2d7e4899d4755ad575feb60c24d4b (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8d8448e41c7842ff8ddf670040a9aebd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.083782 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd2d7e4899d4755ad575feb60c24d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.083899 1517 raft_consensus.cc:482] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:18.084048 1517 raft_consensus.cc:504] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.085059 1517 leader_election.cc:288] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:18.086198 30026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b92831cb328345c28d3fec345b9bc1cd" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:18.086414 30026 raft_consensus.cc:2454] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:18:18.086391 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b92831cb328345c28d3fec345b9bc1cd" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:18:18.086616 27519 raft_consensus.cc:2454] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:18:18.087296 27584 leader_election.cc:302] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:18.087848 1562 tablet_bootstrap.cc:492] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:18.088881 30401 tablet_service.cc:1467] Processing CreateTablet for tablet ecd2d7e4899d4755ad575feb60c24d4b (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8d8448e41c7842ff8ddf670040a9aebd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.089017 1517 raft_consensus.cc:2781] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:18.089130 1517 raft_consensus.cc:482] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:18.089251 1517 raft_consensus.cc:3037] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.089308 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd2d7e4899d4755ad575feb60c24d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.092038 1517 raft_consensus.cc:504] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.093212 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b92831cb328345c28d3fec345b9bc1cd" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:18:18.093407 27519 raft_consensus.cc:3037] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.095996 27519 raft_consensus.cc:2454] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:18.096601 30026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b92831cb328345c28d3fec345b9bc1cd" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:18.096801 30026 raft_consensus.cc:3037] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.097100 27587 leader_election.cc:302] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:18.097354 1314 raft_consensus.cc:2781] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:18.097481 1314 raft_consensus.cc:686] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:18:18.097700 1314 consensus_queue.cc:237] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.099399 30026 raft_consensus.cc:2454] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:18.099445 27370 tablet_service.cc:1467] Processing CreateTablet for tablet d3a3473a93f14baaac48b9b7f47ac2a1 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8d8448e41c7842ff8ddf670040a9aebd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.099956 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a3473a93f14baaac48b9b7f47ac2a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.103298 1517 leader_election.cc:288] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:18.110360 1562 tablet_bootstrap.cc:654] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.112668 27498 tablet_service.cc:1467] Processing CreateTablet for tablet d3a3473a93f14baaac48b9b7f47ac2a1 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8d8448e41c7842ff8ddf670040a9aebd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.113082 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a3473a93f14baaac48b9b7f47ac2a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.112998 30401 tablet_service.cc:1467] Processing CreateTablet for tablet d3a3473a93f14baaac48b9b7f47ac2a1 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=8d8448e41c7842ff8ddf670040a9aebd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.113420 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a3473a93f14baaac48b9b7f47ac2a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.116744 1563 tablet_bootstrap.cc:492] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:18.118273 1563 tablet_bootstrap.cc:654] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.125658 1562 tablet_bootstrap.cc:492] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:18.125742 1564 ts_tablet_manager.cc:1398] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.078s user 0.002s sys 0.000s
I20240502 08:18:18.125845 1562 ts_tablet_manager.cc:1367] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20240502 08:18:18.126778 1562 raft_consensus.cc:348] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:18.127898 1564 tablet_bootstrap.cc:492] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:18.128919 1562 raft_consensus.cc:374] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.129022 1562 raft_consensus.cc:729] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.129346 1562 consensus_queue.cc:260] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:18.130000 1562 ts_tablet_manager.cc:1398] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:18:18.135891 1563 tablet_bootstrap.cc:492] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:18.136114 1563 ts_tablet_manager.cc:1367] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.001s
I20240502 08:18:18.137014 1563 raft_consensus.cc:348] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.137310 1563 raft_consensus.cc:374] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.137427 1563 raft_consensus.cc:729] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.137717 1563 consensus_queue.cc:260] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.138504 1563 ts_tablet_manager.cc:1398] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:18.139531 1583 tablet_bootstrap.cc:492] T d3a3473a93f14baaac48b9b7f47ac2a1 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:18.141548 1583 tablet_bootstrap.cc:654] T d3a3473a93f14baaac48b9b7f47ac2a1 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.142130 1564 tablet_bootstrap.cc:654] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.150197 27110 catalog_manager.cc:5484] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:18.152004 1581 tablet_bootstrap.cc:492] T ecd2d7e4899d4755ad575feb60c24d4b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:18.152036 1583 tablet_bootstrap.cc:492] T d3a3473a93f14baaac48b9b7f47ac2a1 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:18:18.152150 1583 ts_tablet_manager.cc:1367] T d3a3473a93f14baaac48b9b7f47ac2a1 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.005s
I20240502 08:18:18.153179 1583 raft_consensus.cc:348] T d3a3473a93f14baaac48b9b7f47ac2a1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:18.154258 1583 raft_consensus.cc:374] T d3a3473a93f14baaac48b9b7f47ac2a1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.154495 1583 raft_consensus.cc:729] T d3a3473a93f14baaac48b9b7f47ac2a1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.155135 1583 consensus_queue.cc:260] T d3a3473a93f14baaac48b9b7f47ac2a1 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:18.162346 1581 tablet_bootstrap.cc:654] T ecd2d7e4899d4755ad575feb60c24d4b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.167846 1576 tablet_bootstrap.cc:492] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:18.168834 1583 ts_tablet_manager.cc:1398] T d3a3473a93f14baaac48b9b7f47ac2a1 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.016s user 0.000s sys 0.001s
I20240502 08:18:18.170939 1576 tablet_bootstrap.cc:654] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.173452 1581 tablet_bootstrap.cc:492] T ecd2d7e4899d4755ad575feb60c24d4b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:18.173596 1581 ts_tablet_manager.cc:1367] T ecd2d7e4899d4755ad575feb60c24d4b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240502 08:18:18.174242 1581 raft_consensus.cc:348] T ecd2d7e4899d4755ad575feb60c24d4b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.175740 1564 tablet_bootstrap.cc:492] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:18.175979 1564 ts_tablet_manager.cc:1367] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.004s
I20240502 08:18:18.176622 1564 raft_consensus.cc:348] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:18.176867 1564 raft_consensus.cc:374] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.176963 1564 raft_consensus.cc:729] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.177217 1564 consensus_queue.cc:260] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:18.179672 1564 ts_tablet_manager.cc:1398] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:18:18.183614 1564 tablet_bootstrap.cc:492] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:18.186167 1581 raft_consensus.cc:374] T ecd2d7e4899d4755ad575feb60c24d4b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.186265 1581 raft_consensus.cc:729] T ecd2d7e4899d4755ad575feb60c24d4b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.186470 1581 consensus_queue.cc:260] T ecd2d7e4899d4755ad575feb60c24d4b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.187479 1564 tablet_bootstrap.cc:654] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.191253 1581 ts_tablet_manager.cc:1398] T ecd2d7e4899d4755ad575feb60c24d4b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240502 08:18:18.193089 1564 tablet_bootstrap.cc:492] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:18.193230 1564 ts_tablet_manager.cc:1367] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20240502 08:18:18.193939 1564 raft_consensus.cc:348] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:18.194226 1564 raft_consensus.cc:374] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.194365 1564 raft_consensus.cc:729] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.194629 1564 consensus_queue.cc:260] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:18.195364 1564 ts_tablet_manager.cc:1398] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:18.196121 1576 tablet_bootstrap.cc:492] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:18.196331 1576 ts_tablet_manager.cc:1367] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20240502 08:18:18.197232 1576 raft_consensus.cc:348] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:18.197454 1576 raft_consensus.cc:374] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.197575 1576 raft_consensus.cc:729] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.197862 1576 consensus_queue.cc:260] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:18.198617 1576 ts_tablet_manager.cc:1398] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:18.199047 1576 tablet_bootstrap.cc:492] T ecd2d7e4899d4755ad575feb60c24d4b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:18.201210 1576 tablet_bootstrap.cc:654] T ecd2d7e4899d4755ad575feb60c24d4b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.216585 1576 tablet_bootstrap.cc:492] T ecd2d7e4899d4755ad575feb60c24d4b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:18.216749 1576 ts_tablet_manager.cc:1367] T ecd2d7e4899d4755ad575feb60c24d4b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.009s
I20240502 08:18:18.217463 1576 raft_consensus.cc:348] T ecd2d7e4899d4755ad575feb60c24d4b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.217669 1576 raft_consensus.cc:374] T ecd2d7e4899d4755ad575feb60c24d4b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.217739 1576 raft_consensus.cc:729] T ecd2d7e4899d4755ad575feb60c24d4b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.218070 1576 consensus_queue.cc:260] T ecd2d7e4899d4755ad575feb60c24d4b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.218720 1576 ts_tablet_manager.cc:1398] T ecd2d7e4899d4755ad575feb60c24d4b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:18.218984 1576 tablet_bootstrap.cc:492] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:18.220681 1576 tablet_bootstrap.cc:654] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.235637 1576 tablet_bootstrap.cc:492] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:18.235805 1576 ts_tablet_manager.cc:1367] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.001s
I20240502 08:18:18.237260 1576 raft_consensus.cc:348] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:18.237455 1576 raft_consensus.cc:374] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.237550 1576 raft_consensus.cc:729] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.237756 1576 consensus_queue.cc:260] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:18.241997 1576 ts_tablet_manager.cc:1398] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:18:18.255731 1400 raft_consensus.cc:482] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:18.255884 1400 raft_consensus.cc:504] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:18.256525 1400 leader_election.cc:288] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:18.257449 27518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a3473a93f14baaac48b9b7f47ac2a1" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:18:18.257649 27518 raft_consensus.cc:2454] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:18.257982 27648 raft_consensus.cc:1261] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:18:18.258126 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a3473a93f14baaac48b9b7f47ac2a1" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:18:18.258298 27388 raft_consensus.cc:2454] T d3a3473a93f14baaac48b9b7f47ac2a1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:18.258769 1428 consensus_queue.cc:1035] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:18.259232 30357 leader_election.cc:302] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:18.261518 1400 raft_consensus.cc:2781] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:18.261636 1400 raft_consensus.cc:482] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:18.261731 1400 raft_consensus.cc:3037] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.263592 1400 raft_consensus.cc:504] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:18.264047 1400 leader_election.cc:288] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:18.264917 27518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a3473a93f14baaac48b9b7f47ac2a1" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:18:18.265108 27518 raft_consensus.cc:3037] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.266093 1314 raft_consensus.cc:482] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:18.266229 1314 raft_consensus.cc:504] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.266925 1314 leader_election.cc:288] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:18.267638 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd2d7e4899d4755ad575feb60c24d4b" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:18.267822 27255 raft_consensus.cc:2454] T ecd2d7e4899d4755ad575feb60c24d4b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:18:18.268234 27586 leader_election.cc:302] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:18.268226 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a3473a93f14baaac48b9b7f47ac2a1" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:18:18.268399 27388 raft_consensus.cc:3037] T d3a3473a93f14baaac48b9b7f47ac2a1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.268617 1314 raft_consensus.cc:2781] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:18.268707 1314 raft_consensus.cc:482] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:18.268781 1314 raft_consensus.cc:3037] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.270990 27388 raft_consensus.cc:2454] T d3a3473a93f14baaac48b9b7f47ac2a1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:18.271335 1314 raft_consensus.cc:504] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.272446 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd2d7e4899d4755ad575feb60c24d4b" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:18.272596 27255 raft_consensus.cc:3037] T ecd2d7e4899d4755ad575feb60c24d4b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.273309 30290 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "039b1aea12ba4627882b603336070773"
dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56572
I20240502 08:18:18.273576 30290 raft_consensus.cc:595] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:18:18.275019 27255 raft_consensus.cc:2454] T ecd2d7e4899d4755ad575feb60c24d4b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:18.275329 27115 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:18.276137 27586 leader_election.cc:302] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:18.276402 1502 raft_consensus.cc:2781] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:18.276518 1502 raft_consensus.cc:686] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:18:18.276867 1314 leader_election.cc:288] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:18.276751 1502 consensus_queue.cc:237] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.276885 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd2d7e4899d4755ad575feb60c24d4b" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:18.277060 30423 raft_consensus.cc:2454] T ecd2d7e4899d4755ad575feb60c24d4b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:18:18.277381 30419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd2d7e4899d4755ad575feb60c24d4b" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:18.277577 30419 raft_consensus.cc:3037] T ecd2d7e4899d4755ad575feb60c24d4b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.280373 30419 raft_consensus.cc:2454] T ecd2d7e4899d4755ad575feb60c24d4b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:18.285118 30357 leader_election.cc:302] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:18.286509 1400 raft_consensus.cc:2781] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:18.286645 1400 raft_consensus.cc:686] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:18:18.286885 1400 consensus_queue.cc:237] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:18.290670 27519 raft_consensus.cc:1261] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:18.291357 1517 consensus_queue.cc:1035] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.292596 30025 raft_consensus.cc:1261] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:18.296497 1314 consensus_queue.cc:1035] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.304436 1428 raft_consensus.cc:482] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:18.304651 1428 raft_consensus.cc:504] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:18.305429 1428 leader_election.cc:288] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:18.306372 30419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d122f68d36024614a017e8cc3a1224c1" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:18.306571 30419 raft_consensus.cc:2454] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:18:18.306644 30026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d122f68d36024614a017e8cc3a1224c1" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:18.306859 30026 raft_consensus.cc:2454] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:18:18.307039 27453 leader_election.cc:302] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:18.307561 1428 raft_consensus.cc:2781] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:18.307634 1428 raft_consensus.cc:482] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:18.307703 1428 raft_consensus.cc:3037] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.313731 1428 raft_consensus.cc:504] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:18.317687 27115 catalog_manager.cc:5484] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:18.323988 27110 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet b92831cb328345c28d3fec345b9bc1cd (table rebalancer_dynamic_table [id=c7bc9125709d40d2b0ba8a270d9175c1]). Expected version 1 got 0
I20240502 08:18:18.324427 30419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d122f68d36024614a017e8cc3a1224c1" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:18.324643 30419 raft_consensus.cc:3037] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.325045 30026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d122f68d36024614a017e8cc3a1224c1" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:18.325224 30026 raft_consensus.cc:3037] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.325320 27518 raft_consensus.cc:2454] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:18.326893 1428 leader_election.cc:288] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:18.327726 30026 raft_consensus.cc:2454] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:18:18.328121 27453 leader_election.cc:302] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:18.328146 30419 raft_consensus.cc:2454] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:18:18.328354 1428 raft_consensus.cc:2781] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:18.328456 1428 raft_consensus.cc:686] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:18:18.328756 1428 consensus_queue.cc:237] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:18.331444 27115 catalog_manager.cc:5484] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:18.347862 1610 tablet.cc:1722] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:18.348697 1610 tablet.cc:1620] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442: MemRowSet was empty: no flush needed.
I20240502 08:18:18.351945 30156 raft_consensus.cc:1261] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:18:18.352331 27110 catalog_manager.cc:5576] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469) does not have the latest schema for tablet b92831cb328345c28d3fec345b9bc1cd (table rebalancer_dynamic_table [id=c7bc9125709d40d2b0ba8a270d9175c1]). Expected version 1 got 0
I20240502 08:18:18.352360 1611 tablet.cc:1722] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:18.352433 1185 consensus_queue.cc:1035] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:18.352854 1611 tablet.cc:1620] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:18.355911 27115 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=c7bc9125709d40d2b0ba8a270d9175c1] alter complete (version 1)
I20240502 08:18:18.360492 30419 raft_consensus.cc:1261] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:18:18.361150 1185 consensus_queue.cc:1035] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:18.364537 27110 catalog_manager.cc:5484] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:18.365710 27109 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet b92831cb328345c28d3fec345b9bc1cd (table rebalancer_dynamic_table [id=c7bc9125709d40d2b0ba8a270d9175c1]). Expected version 1 got 0
I20240502 08:18:18.385681 27518 consensus_queue.cc:237] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:18.390326 30157 raft_consensus.cc:1261] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:18:18.391008 27648 raft_consensus.cc:1261] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:18:18.392019 1185 consensus_queue.cc:1035] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:18.392220 1428 consensus_queue.cc:1035] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:18.397783 1185 raft_consensus.cc:2932] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:18.401410 30157 raft_consensus.cc:2932] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:18.406512 27650 raft_consensus.cc:2932] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:18.408805 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fefcc470084f4ec9be6ba45d3ba22ce5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:18.413529 27110 catalog_manager.cc:5484] T fefcc470084f4ec9be6ba45d3ba22ce5 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 3 to 5, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New cstate: current_term: 2 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:18.421450 30400 tablet_service.cc:1514] Processing DeleteTablet for tablet fefcc470084f4ec9be6ba45d3ba22ce5 with delete_type TABLET_DATA_TOMBSTONED (TS 31a1e33dcdbf42f58ea72ce1c0afb0ad not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56694
W20240502 08:18:18.424129 1609 log.cc:924] Time spent T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20240502 08:18:18.462046 1618 tablet.cc:1722] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:18.463682 1618 tablet.cc:1620] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59: MemRowSet was empty: no flush needed.
I20240502 08:18:18.469379 27110 catalog_manager.cc:5576] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469) does not have the latest schema for tablet b92831cb328345c28d3fec345b9bc1cd (table rebalancer_dynamic_table [id=c7bc9125709d40d2b0ba8a270d9175c1]). Expected version 1 got 0
I20240502 08:18:18.473816 27115 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:18.483978 1617 tablet_replica.cc:330] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:18.484267 1617 raft_consensus.cc:2229] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:18.490536 1610 tablet.cc:1722] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:18.491024 1610 tablet.cc:1620] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442: MemRowSet was empty: no flush needed.
I20240502 08:18:18.491465 1611 tablet.cc:1722] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:18.491884 1611 tablet.cc:1620] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:18.501407 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=c7bc9125709d40d2b0ba8a270d9175c1] alter complete (version 2)
I20240502 08:18:18.504135 1617 pending_rounds.cc:62] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Trying to abort 1 pending ops.
I20240502 08:18:18.504254 1617 pending_rounds.cc:68] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Aborting op as it isn't in flight: id { term: 2 index: 4 } timestamp: 7023156829265211392 op_type: NO_OP noop_request { }
I20240502 08:18:18.509867 1617 raft_consensus.cc:2866] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:18:18.510046 1617 raft_consensus.cc:2258] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:18.510993 1617 ts_tablet_manager.cc:1875] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:18.517556 1617 ts_tablet_manager.cc:1888] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 08:18:18.517725 1617 log.cc:1196] T fefcc470084f4ec9be6ba45d3ba22ce5 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/fefcc470084f4ec9be6ba45d3ba22ce5
I20240502 08:18:18.518569 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet fefcc470084f4ec9be6ba45d3ba22ce5 (table rebalancer_extra_table_30 [id=8fad8146798f48ce9901b24b7f1944b1]) successfully deleted
I20240502 08:18:18.520792 27115 catalog_manager.cc:5576] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469) does not have the latest schema for tablet b92831cb328345c28d3fec345b9bc1cd (table rebalancer_dynamic_table [id=c7bc9125709d40d2b0ba8a270d9175c1]). Expected version 2 got 1
I20240502 08:18:18.525780 1618 tablet.cc:1722] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:18.526312 1618 tablet.cc:1620] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59: MemRowSet was empty: no flush needed.
I20240502 08:18:18.567451 30025 consensus_queue.cc:237] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:18.569756 27650 raft_consensus.cc:1261] T d00f25f29f5d4db2a2261dd37c0793fa P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:18.569880 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d00f25f29f5d4db2a2261dd37c0793fa. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:18.570259 30157 raft_consensus.cc:1261] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:18.570355 1476 consensus_queue.cc:1035] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:18.570956 1129 consensus_queue.cc:1035] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:18.573827 1474 raft_consensus.cc:2932] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:18.574355 27650 raft_consensus.cc:2932] T d00f25f29f5d4db2a2261dd37c0793fa P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:18.580672 27115 catalog_manager.cc:5484] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:18.588804 30157 raft_consensus.cc:2932] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:18.594704 623 raft_consensus.cc:979] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4: : Instructing follower 1f0bf4bc3c5146539146b825b048bff3 to start an election
I20240502 08:18:18.594853 623 raft_consensus.cc:1067] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Signalling peer 1f0bf4bc3c5146539146b825b048bff3 to start an election
I20240502 08:18:18.595819 30025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "039b1aea12ba4627882b603336070773"
dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
from {username='slave'} at 127.20.207.7:60733
I20240502 08:18:18.596081 30025 raft_consensus.cc:482] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:18.596189 30025 raft_consensus.cc:3037] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:18.599498 30025 raft_consensus.cc:504] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:18.600816 30025 leader_election.cc:288] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:18.601032 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039b1aea12ba4627882b603336070773" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:18.601231 27255 raft_consensus.cc:3037] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:18.604602 27255 raft_consensus.cc:2454] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 2.
I20240502 08:18:18.604933 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:18.606798 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039b1aea12ba4627882b603336070773" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:18.606997 27650 raft_consensus.cc:3037] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:18.608134 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039b1aea12ba4627882b603336070773" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:18.608422 30290 raft_consensus.cc:3032] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:18.608525 30290 raft_consensus.cc:729] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:18:18.608904 30290 consensus_queue.cc:260] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:18.609505 30290 raft_consensus.cc:3037] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:18.609637 27650 raft_consensus.cc:2454] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 2.
I20240502 08:18:18.610204 29964 leader_election.cc:302] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 1f0bf4bc3c5146539146b825b048bff3, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:18.610518 1129 raft_consensus.cc:2781] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:18.610721 1129 raft_consensus.cc:686] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:18:18.611075 1129 consensus_queue.cc:237] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:18.618912 1610 tablet.cc:1722] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:18.620092 1618 tablet.cc:1722] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:18.625301 27115 catalog_manager.cc:5484] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 1 to 2, leader changed from 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 2 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:18.627835 1611 tablet.cc:1722] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:18.628578 30290 raft_consensus.cc:2454] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 2.
I20240502 08:18:18.633327 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=c7bc9125709d40d2b0ba8a270d9175c1] alter complete (version 3)
I20240502 08:18:18.650365 27519 consensus_queue.cc:237] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:18.653262 30419 raft_consensus.cc:1261] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:18.653414 30025 raft_consensus.cc:1261] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:18.653625 27456 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 -> Peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Couldn't send request to peer bf8ac71b15f740a0a928911360773442. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2abc11a238a542e4982452b5ce341101. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:18.653868 1428 consensus_queue.cc:1035] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:18.654201 1185 consensus_queue.cc:1035] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:18.656560 1428 raft_consensus.cc:2932] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:18.657763 1502 consensus_queue.cc:1035] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.658032 30419 raft_consensus.cc:2932] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:18.669296 30025 raft_consensus.cc:2932] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:18.680042 27110 catalog_manager.cc:5484] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:18.704725 1502 consensus_queue.cc:1035] T ecd2d7e4899d4755ad575feb60c24d4b P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.727380 30290 consensus_queue.cc:237] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:18.735165 30423 raft_consensus.cc:1261] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:18.735441 27255 raft_consensus.cc:1261] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:18.735749 1661 consensus_queue.cc:1035] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:18.736114 623 consensus_queue.cc:1035] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:18.736367 30225 consensus_peers.cc:485] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2066c7aa052342a1aca3eb849aa251fc. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:18.739136 623 raft_consensus.cc:2932] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:18.743059 30423 raft_consensus.cc:2932] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:18.744935 27109 catalog_manager.cc:5484] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:18.746558 27255 raft_consensus.cc:2932] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:18.749537 27110 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:18.749728 27110 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:18.752847 27110 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet b92831cb328345c28d3fec345b9bc1cd
I20240502 08:18:18.753919 27498 tablet_service.cc:1514] Processing DeleteTablet for tablet b92831cb328345c28d3fec345b9bc1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:18 UTC) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:18.754132 27629 tablet_service.cc:1514] Processing DeleteTablet for tablet b92831cb328345c28d3fec345b9bc1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:18 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:18:18.755363 30007 tablet_service.cc:1514] Processing DeleteTablet for tablet b92831cb328345c28d3fec345b9bc1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:18 UTC) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:18.780143 1191 consensus_queue.cc:1035] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.781091 1665 tablet_replica.cc:330] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:18.781375 1665 raft_consensus.cc:2229] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:18.781620 1665 raft_consensus.cc:2258] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:18.784051 1665 ts_tablet_manager.cc:1875] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:18.790195 1667 tablet_replica.cc:330] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:18.790613 1666 tablet_replica.cc:330] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:18:18.790867 1667 raft_consensus.cc:2229] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:18.791298 1666 raft_consensus.cc:2229] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:18.791545 1667 raft_consensus.cc:2258] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:18.792672 1665 ts_tablet_manager.cc:1888] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:18.792970 1667 ts_tablet_manager.cc:1875] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:18.793490 1666 raft_consensus.cc:2258] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:18.796113 1666 ts_tablet_manager.cc:1875] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:18.802664 1400 consensus_queue.cc:1035] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.809088 1665 log.cc:1196] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/b92831cb328345c28d3fec345b9bc1cd
I20240502 08:18:18.809728 1665 ts_tablet_manager.cc:1909] T b92831cb328345c28d3fec345b9bc1cd P b01e7c8a302d41078fa39cfef665ff59: Deleting consensus metadata
I20240502 08:18:18.812461 1666 ts_tablet_manager.cc:1888] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:18.813179 1667 ts_tablet_manager.cc:1888] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:18.813867 1667 log.cc:1196] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/b92831cb328345c28d3fec345b9bc1cd
I20240502 08:18:18.813684 1666 log.cc:1196] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/b92831cb328345c28d3fec345b9bc1cd
I20240502 08:18:18.814626 1667 ts_tablet_manager.cc:1909] T b92831cb328345c28d3fec345b9bc1cd P 1f0bf4bc3c5146539146b825b048bff3: Deleting consensus metadata
I20240502 08:18:18.815241 1666 ts_tablet_manager.cc:1909] T b92831cb328345c28d3fec345b9bc1cd P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:18:18.816448 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet b92831cb328345c28d3fec345b9bc1cd (table rebalancer_dynamic_table_new_name [id=c7bc9125709d40d2b0ba8a270d9175c1]) successfully deleted
I20240502 08:18:18.816767 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet b92831cb328345c28d3fec345b9bc1cd (table rebalancer_dynamic_table_new_name [id=c7bc9125709d40d2b0ba8a270d9175c1]) successfully deleted
I20240502 08:18:18.818150 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet b92831cb328345c28d3fec345b9bc1cd (table rebalancer_dynamic_table_new_name [id=c7bc9125709d40d2b0ba8a270d9175c1]) successfully deleted
I20240502 08:18:18.823751 1650 consensus_queue.cc:1035] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.838980 1650 consensus_queue.cc:1035] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.858886 27110 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:18.869267 30270 tablet_service.cc:1467] Processing CreateTablet for tablet ceb53dbc8fd54fa8add21d1bea0c5fc5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=caf64b278d8842a39263b02f0a0c8a51]), partition=
I20240502 08:18:18.869647 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb53dbc8fd54fa8add21d1bea0c5fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.869663 27238 tablet_service.cc:1467] Processing CreateTablet for tablet ceb53dbc8fd54fa8add21d1bea0c5fc5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=caf64b278d8842a39263b02f0a0c8a51]), partition=
I20240502 08:18:18.869969 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb53dbc8fd54fa8add21d1bea0c5fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.874661 30007 tablet_service.cc:1467] Processing CreateTablet for tablet ceb53dbc8fd54fa8add21d1bea0c5fc5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=caf64b278d8842a39263b02f0a0c8a51]), partition=
I20240502 08:18:18.874994 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb53dbc8fd54fa8add21d1bea0c5fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.890822 27115 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:18.891692 1679 tablet_bootstrap.cc:492] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:18.896016 1679 tablet_bootstrap.cc:654] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.896481 27388 consensus_queue.cc:237] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:18.900833 1681 tablet_bootstrap.cc:492] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:18.900928 27370 tablet_service.cc:1467] Processing CreateTablet for tablet bc8fe72edcef41a28fdeea86d501f93b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=327af2c89b2f41028a116c0587a226ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.900960 30138 tablet_service.cc:1467] Processing CreateTablet for tablet 459e9f5bf6d24ba9b36686d49ff12e2b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=327af2c89b2f41028a116c0587a226ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.901300 27370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc8fe72edcef41a28fdeea86d501f93b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.901352 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 459e9f5bf6d24ba9b36686d49ff12e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.902776 30006 tablet_service.cc:1467] Processing CreateTablet for tablet e80b9a98bdd041b1ac97af055951a92a (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=327af2c89b2f41028a116c0587a226ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.903189 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e80b9a98bdd041b1ac97af055951a92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.904819 30400 tablet_service.cc:1467] Processing CreateTablet for tablet bc8fe72edcef41a28fdeea86d501f93b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=327af2c89b2f41028a116c0587a226ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.905239 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc8fe72edcef41a28fdeea86d501f93b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.905325 27629 tablet_service.cc:1467] Processing CreateTablet for tablet 459e9f5bf6d24ba9b36686d49ff12e2b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=327af2c89b2f41028a116c0587a226ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.905710 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 459e9f5bf6d24ba9b36686d49ff12e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.906872 27498 tablet_service.cc:1467] Processing CreateTablet for tablet e80b9a98bdd041b1ac97af055951a92a (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=327af2c89b2f41028a116c0587a226ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.907228 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e80b9a98bdd041b1ac97af055951a92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.913937 27237 tablet_service.cc:1467] Processing CreateTablet for tablet e80b9a98bdd041b1ac97af055951a92a (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=327af2c89b2f41028a116c0587a226ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.914350 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e80b9a98bdd041b1ac97af055951a92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.920768 27366 tablet_service.cc:1467] Processing CreateTablet for tablet 459e9f5bf6d24ba9b36686d49ff12e2b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=327af2c89b2f41028a116c0587a226ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.921118 27366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 459e9f5bf6d24ba9b36686d49ff12e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.932562 27238 tablet_service.cc:1467] Processing CreateTablet for tablet bc8fe72edcef41a28fdeea86d501f93b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=327af2c89b2f41028a116c0587a226ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.933128 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc8fe72edcef41a28fdeea86d501f93b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.933336 1680 tablet_bootstrap.cc:492] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:18.934995 1681 tablet_bootstrap.cc:654] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.937634 1680 tablet_bootstrap.cc:654] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.940279 1684 tablet_bootstrap.cc:492] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:18.941679 1681 tablet_bootstrap.cc:492] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:18.942409 1684 tablet_bootstrap.cc:654] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.943070 1681 ts_tablet_manager.cc:1367] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20240502 08:18:18.944483 1681 raft_consensus.cc:348] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:18.946887 1681 raft_consensus.cc:374] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.947471 1681 raft_consensus.cc:729] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.947813 1689 tablet_bootstrap.cc:492] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:18.948359 1681 consensus_queue.cc:260] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:18.952199 1681 ts_tablet_manager.cc:1398] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 08:18:18.954159 1689 tablet_bootstrap.cc:654] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.954557 1690 tablet_bootstrap.cc:492] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:18.956761 1690 tablet_bootstrap.cc:654] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.976570 30423 raft_consensus.cc:1261] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:18:18.976852 27326 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Couldn't send request to peer 84cda20738d2452da166e31fc6da57a6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04a0a58eb52f41bf818e5102996d09e0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:18.977171 27255 raft_consensus.cc:1261] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:18.977591 1692 tablet_bootstrap.cc:492] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:18.978250 1678 consensus_queue.cc:1035] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:18.979560 1692 tablet_bootstrap.cc:654] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.981062 1685 tablet_bootstrap.cc:492] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:18.981431 1678 consensus_queue.cc:1035] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:18.983362 1685 tablet_bootstrap.cc:654] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.984673 1689 tablet_bootstrap.cc:492] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:18.984915 1689 ts_tablet_manager.cc:1367] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.002s
I20240502 08:18:18.985667 1689 raft_consensus.cc:348] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:18.989487 1680 tablet_bootstrap.cc:492] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:18.989647 1680 ts_tablet_manager.cc:1367] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.086s user 0.000s sys 0.011s
I20240502 08:18:18.990553 1680 raft_consensus.cc:348] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:18.990834 1680 raft_consensus.cc:374] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.990952 1689 raft_consensus.cc:374] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.991216 1689 raft_consensus.cc:729] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.991313 1679 tablet_bootstrap.cc:492] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:18.991099 1680 raft_consensus.cc:729] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.991533 1679 ts_tablet_manager.cc:1367] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.100s user 0.006s sys 0.000s
I20240502 08:18:18.991578 1689 consensus_queue.cc:260] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:18.991998 1680 consensus_queue.cc:260] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:18.992331 1679 raft_consensus.cc:348] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:18.992566 1679 raft_consensus.cc:374] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.992647 1689 ts_tablet_manager.cc:1398] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 08:18:18.992679 1679 raft_consensus.cc:729] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.992950 1679 consensus_queue.cc:260] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:18.993145 1680 ts_tablet_manager.cc:1398] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 08:18:18.993419 1680 tablet_bootstrap.cc:492] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:18.994756 1684 tablet_bootstrap.cc:492] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:18.995019 1684 ts_tablet_manager.cc:1367] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.055s user 0.010s sys 0.000s
I20240502 08:18:18.995841 1684 raft_consensus.cc:348] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:18.996173 1684 raft_consensus.cc:374] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.996318 1684 raft_consensus.cc:729] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.997020 1684 consensus_queue.cc:260] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:19.001060 1684 ts_tablet_manager.cc:1398] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:18:19.002804 1679 ts_tablet_manager.cc:1398] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240502 08:18:19.002839 1680 tablet_bootstrap.cc:654] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.006798 1679 tablet_bootstrap.cc:492] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:19.008332 1679 tablet_bootstrap.cc:654] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.009259 1685 tablet_bootstrap.cc:492] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:19.009450 1685 ts_tablet_manager.cc:1367] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20240502 08:18:19.011372 1685 raft_consensus.cc:348] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:19.011689 1685 raft_consensus.cc:374] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.012197 1680 tablet_bootstrap.cc:492] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:19.012405 1680 ts_tablet_manager.cc:1367] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
I20240502 08:18:19.013017 1685 raft_consensus.cc:729] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.013211 1680 raft_consensus.cc:348] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:19.013494 1680 raft_consensus.cc:374] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.013610 1680 raft_consensus.cc:729] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.013693 1679 tablet_bootstrap.cc:492] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:19.013890 1680 consensus_queue.cc:260] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:19.013939 1685 consensus_queue.cc:260] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:19.016219 1680 ts_tablet_manager.cc:1398] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:19.017436 1690 tablet_bootstrap.cc:492] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:19.017590 1690 ts_tablet_manager.cc:1367] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.005s
I20240502 08:18:19.017530 1695 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:19.018368 1690 raft_consensus.cc:348] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:19.018605 1690 raft_consensus.cc:374] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.018687 1690 raft_consensus.cc:729] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.018961 1690 consensus_queue.cc:260] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:19.019551 1690 ts_tablet_manager.cc:1398] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:19.020010 1694 ts_tablet_manager.cc:902] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59: Initiating tablet copy from peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:19.020208 1679 ts_tablet_manager.cc:1367] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 08:18:19.021008 1679 raft_consensus.cc:348] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:19.022508 1692 tablet_bootstrap.cc:492] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:18:19.022684 1692 ts_tablet_manager.cc:1367] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20240502 08:18:19.023380 1692 raft_consensus.cc:348] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:19.023607 1692 raft_consensus.cc:374] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.023702 1692 raft_consensus.cc:729] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.023952 1692 consensus_queue.cc:260] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:19.024495 1692 ts_tablet_manager.cc:1398] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:19.020021 1685 ts_tablet_manager.cc:1398] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 08:18:19.028473 27258 raft_consensus.cc:1261] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:18:19.029615 1129 consensus_queue.cc:1035] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:19.030951 27257 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:19.040832 1679 raft_consensus.cc:374] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.041126 1679 raft_consensus.cc:729] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.042157 27109 catalog_manager.cc:5484] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:19.043255 30423 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:19.041791 1679 consensus_queue.cc:260] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:19.053608 1679 ts_tablet_manager.cc:1398] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20240502 08:18:19.054682 1679 tablet_bootstrap.cc:492] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:19.055047 1692 tablet_bootstrap.cc:492] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:19.056792 1692 tablet_bootstrap.cc:654] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.059727 1679 tablet_bootstrap.cc:654] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.080550 1692 tablet_bootstrap.cc:492] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:18:19.080762 1692 ts_tablet_manager.cc:1367] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.018s
I20240502 08:18:19.081768 1692 raft_consensus.cc:348] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:19.082239 1692 raft_consensus.cc:374] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.082396 1692 raft_consensus.cc:729] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.082896 1679 tablet_bootstrap.cc:492] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:19.082854 1692 consensus_queue.cc:260] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:19.083062 1679 ts_tablet_manager.cc:1367] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20240502 08:18:19.083779 1679 raft_consensus.cc:348] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:19.084012 1679 raft_consensus.cc:374] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.084096 1679 raft_consensus.cc:729] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.084333 1679 consensus_queue.cc:260] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:19.084952 1679 ts_tablet_manager.cc:1398] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:19.085593 1692 ts_tablet_manager.cc:1398] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:18:19.089679 27650 raft_consensus.cc:1261] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:18:19.091041 1129 raft_consensus.cc:482] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:19.091210 1129 raft_consensus.cc:504] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:19.091728 1129 leader_election.cc:288] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:19.092793 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb53dbc8fd54fa8add21d1bea0c5fc5" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:19.092975 27258 raft_consensus.cc:2454] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:19.093524 1476 consensus_queue.cc:1035] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:19.095536 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb53dbc8fd54fa8add21d1bea0c5fc5" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:19.095779 30290 raft_consensus.cc:2454] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 0.
I20240502 08:18:19.095830 29963 leader_election.cc:302] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 1f0bf4bc3c5146539146b825b048bff3; no voters:
I20240502 08:18:19.096194 1476 raft_consensus.cc:2781] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:19.096297 1476 raft_consensus.cc:482] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:19.096400 1476 raft_consensus.cc:3037] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.101256 1476 raft_consensus.cc:504] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:19.104290 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb53dbc8fd54fa8add21d1bea0c5fc5" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:19.104521 30290 raft_consensus.cc:3037] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.108390 30290 raft_consensus.cc:2454] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:19.109939 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb53dbc8fd54fa8add21d1bea0c5fc5" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:19.110193 27258 raft_consensus.cc:3037] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.116794 27258 raft_consensus.cc:2454] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:19.117367 29963 leader_election.cc:302] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 1f0bf4bc3c5146539146b825b048bff3; no voters:
I20240502 08:18:19.117693 1129 raft_consensus.cc:2781] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:19.109694 1476 leader_election.cc:288] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:19.123829 1129 raft_consensus.cc:686] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:18:19.124241 1129 consensus_queue.cc:237] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:19.141983 27116 catalog_manager.cc:5484] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:19.145540 1694 tablet_copy_client.cc:308] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.5:37815
I20240502 08:18:19.146870 30290 raft_consensus.cc:1261] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:18:19.147954 30048 tablet_copy_service.cc:140] P 1f0bf4bc3c5146539146b825b048bff3: Received BeginTabletCopySession request for tablet d00f25f29f5d4db2a2261dd37c0793fa from peer b01e7c8a302d41078fa39cfef665ff59 ({username='slave'} at 127.20.207.3:54281)
I20240502 08:18:19.148113 30048 tablet_copy_service.cc:161] P 1f0bf4bc3c5146539146b825b048bff3: Beginning new tablet copy session on tablet d00f25f29f5d4db2a2261dd37c0793fa from peer b01e7c8a302d41078fa39cfef665ff59 at {username='slave'} at 127.20.207.3:54281: session id = b01e7c8a302d41078fa39cfef665ff59-d00f25f29f5d4db2a2261dd37c0793fa
I20240502 08:18:19.150310 1476 consensus_queue.cc:1035] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:19.151041 30048 tablet_copy_source_session.cc:205] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:19.153321 1694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00f25f29f5d4db2a2261dd37c0793fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.164073 1694 tablet_copy_client.cc:783] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:19.164374 1694 tablet_copy_client.cc:647] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:19.171365 30026 consensus_queue.cc:237] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:19.172434 1694 tablet_copy_client.cc:523] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:19.174190 27650 raft_consensus.cc:1261] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:18:19.175410 1476 consensus_queue.cc:1035] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:19.175621 1694 tablet_bootstrap.cc:492] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:19.176517 27257 raft_consensus.cc:1261] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:18:19.179168 1476 consensus_queue.cc:1035] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:19.183425 1712 raft_consensus.cc:2932] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:19.184019 27648 raft_consensus.cc:2932] T 039b1aea12ba4627882b603336070773 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:19.186550 1650 raft_consensus.cc:482] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:19.186673 1650 raft_consensus.cc:504] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:19.187319 1650 leader_election.cc:288] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:19.187821 27256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e80b9a98bdd041b1ac97af055951a92a" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:19.188009 27256 raft_consensus.cc:2454] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:18:19.188570 30026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e80b9a98bdd041b1ac97af055951a92a" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:19.188715 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 039b1aea12ba4627882b603336070773 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:19.188817 30026 raft_consensus.cc:2454] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:18:19.189220 27453 leader_election.cc:302] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:19.189121 27516 consensus_queue.cc:237] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:19.189442 1650 raft_consensus.cc:2781] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:19.189503 1650 raft_consensus.cc:482] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:19.189556 1650 raft_consensus.cc:3037] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.191093 27257 raft_consensus.cc:2932] T 039b1aea12ba4627882b603336070773 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:19.192174 1650 raft_consensus.cc:504] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:19.192641 1650 leader_election.cc:288] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:19.193475 27256 raft_consensus.cc:1261] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:19.193435 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e80b9a98bdd041b1ac97af055951a92a" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:19.193599 27255 raft_consensus.cc:3037] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.194196 27650 raft_consensus.cc:1261] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:18:19.194521 27453 consensus_peers.cc:485] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fef4f56761bb458c8d84807feb38a617. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:19.196239 27255 raft_consensus.cc:2454] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:18:19.198290 1647 consensus_queue.cc:1035] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:19.198735 27115 catalog_manager.cc:5484] T 039b1aea12ba4627882b603336070773 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 3 to 5, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New cstate: current_term: 2 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:19.197925 27455 leader_election.cc:302] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:19.202050 30026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e80b9a98bdd041b1ac97af055951a92a" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:19.202234 30026 raft_consensus.cc:3037] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.204042 1541 raft_consensus.cc:482] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:19.204192 1541 raft_consensus.cc:504] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:19.204878 1541 leader_election.cc:288] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:19.205667 30156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "459e9f5bf6d24ba9b36686d49ff12e2b" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:19.205901 30156 raft_consensus.cc:2454] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:18:19.206576 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet 039b1aea12ba4627882b603336070773 with delete_type TABLET_DATA_TOMBSTONED (TS 49c5c8f70e494edd9fcda40cc0af6cc4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:19.206629 27326 leader_election.cc:302] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:19.207036 1541 raft_consensus.cc:2781] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:19.207145 1541 raft_consensus.cc:482] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:19.207229 1541 raft_consensus.cc:3037] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.198881 1647 consensus_queue.cc:1035] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20240502 08:18:19.201051 1428 raft_consensus.cc:2781] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:19.209611 1428 raft_consensus.cc:686] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:18:19.209918 1428 consensus_queue.cc:237] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:19.210659 1185 raft_consensus.cc:2932] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:19.211081 1541 raft_consensus.cc:504] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:19.212064 30156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "459e9f5bf6d24ba9b36686d49ff12e2b" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:19.212249 30156 raft_consensus.cc:3037] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.214856 30156 raft_consensus.cc:2454] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:18:19.215327 27326 leader_election.cc:302] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:19.215621 1704 raft_consensus.cc:2781] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:19.215767 1704 raft_consensus.cc:686] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:18:19.215767 27650 raft_consensus.cc:2932] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:19.216048 1704 consensus_queue.cc:237] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:19.217880 27648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "459e9f5bf6d24ba9b36686d49ff12e2b" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:19.218151 27648 raft_consensus.cc:2454] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:18:19.220099 30026 raft_consensus.cc:2454] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:18:19.221884 27648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "459e9f5bf6d24ba9b36686d49ff12e2b" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:19.222222 27648 raft_consensus.cc:3037] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.223634 1541 leader_election.cc:288] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:19.223587 27116 catalog_manager.cc:5484] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:19.232209 27648 raft_consensus.cc:2454] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:18:19.234766 1724 tablet_replica.cc:330] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:19.234988 1724 raft_consensus.cc:2229] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:19.235257 1724 raft_consensus.cc:2258] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:19.239495 1724 ts_tablet_manager.cc:1875] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:19.198509 1694 tablet_bootstrap.cc:492] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:19.241235 1694 tablet_bootstrap.cc:492] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59: Bootstrap complete.
I20240502 08:18:19.241492 1694 ts_tablet_manager.cc:1367] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.000s
I20240502 08:18:19.242163 1694 raft_consensus.cc:348] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:19.242544 1694 raft_consensus.cc:729] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: LEARNER
I20240502 08:18:19.242770 1694 consensus_queue.cc:260] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:19.243391 1694 ts_tablet_manager.cc:1398] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:19.244271 30048 tablet_copy_service.cc:342] P 1f0bf4bc3c5146539146b825b048bff3: Request end of tablet copy session b01e7c8a302d41078fa39cfef665ff59-d00f25f29f5d4db2a2261dd37c0793fa received from {username='slave'} at 127.20.207.3:54281
I20240502 08:18:19.244402 30048 tablet_copy_service.cc:433] P 1f0bf4bc3c5146539146b825b048bff3: ending tablet copy session b01e7c8a302d41078fa39cfef665ff59-d00f25f29f5d4db2a2261dd37c0793fa on tablet d00f25f29f5d4db2a2261dd37c0793fa with peer b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:19.251386 1541 raft_consensus.cc:482] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:19.251572 1541 raft_consensus.cc:504] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:19.252678 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8fe72edcef41a28fdeea86d501f93b" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:19.252919 30423 raft_consensus.cc:2454] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:18:19.253404 27323 leader_election.cc:302] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:19.253749 1695 raft_consensus.cc:2781] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:19.253913 1695 raft_consensus.cc:482] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:19.254089 1695 raft_consensus.cc:3037] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.254412 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8fe72edcef41a28fdeea86d501f93b" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:19.254616 27255 raft_consensus.cc:2454] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 0.
I20240502 08:18:19.255299 1541 leader_election.cc:288] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:19.256835 1695 raft_consensus.cc:504] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:19.257723 1695 leader_election.cc:288] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:19.257773 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8fe72edcef41a28fdeea86d501f93b" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:19.257977 30423 raft_consensus.cc:3037] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.260378 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8fe72edcef41a28fdeea86d501f93b" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:19.260612 27255 raft_consensus.cc:3037] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.260900 30423 raft_consensus.cc:2454] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:18:19.261349 27323 leader_election.cc:302] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:19.262162 1695 raft_consensus.cc:2781] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:19.262391 1695 raft_consensus.cc:686] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:18:19.262748 1695 consensus_queue.cc:237] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:19.265722 27255 raft_consensus.cc:2454] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:18:19.267100 1724 ts_tablet_manager.cc:1888] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 08:18:19.267246 1724 log.cc:1196] T 039b1aea12ba4627882b603336070773 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/039b1aea12ba4627882b603336070773
I20240502 08:18:19.267180 27257 raft_consensus.cc:2932] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:19.268046 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet 039b1aea12ba4627882b603336070773 (table rebalancer_extra_table_22 [id=77ecd1036c454af293decc2ea5173f97]) successfully deleted
I20240502 08:18:19.269721 1731 ts_tablet_manager.cc:902] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:19.272887 27116 catalog_manager.cc:5484] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:19.273517 27115 catalog_manager.cc:5484] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:19.280365 27116 catalog_manager.cc:5484] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:19.292461 1731 tablet_copy_client.cc:308] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:19.293053 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet 2abc11a238a542e4982452b5ce341101 from peer bf8ac71b15f740a0a928911360773442 ({username='slave'} at 127.20.207.4:32927)
I20240502 08:18:19.293192 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet 2abc11a238a542e4982452b5ce341101 from peer bf8ac71b15f740a0a928911360773442 at {username='slave'} at 127.20.207.4:32927: session id = bf8ac71b15f740a0a928911360773442-2abc11a238a542e4982452b5ce341101
I20240502 08:18:19.296187 27541 tablet_copy_source_session.cc:205] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:19.300175 1731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2abc11a238a542e4982452b5ce341101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.303622 27116 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:19.309798 27257 raft_consensus.cc:1261] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:19.310680 30290 raft_consensus.cc:1261] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:19.310940 1129 consensus_queue.cc:1035] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.311194 1712 consensus_queue.cc:1035] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.317178 27519 consensus_queue.cc:237] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:19.327690 27116 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet ceb53dbc8fd54fa8add21d1bea0c5fc5 (table rebalancer_dynamic_table [id=caf64b278d8842a39263b02f0a0c8a51]). Expected version 1 got 0
I20240502 08:18:19.344519 27116 catalog_manager.cc:5576] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505) does not have the latest schema for tablet ceb53dbc8fd54fa8add21d1bea0c5fc5 (table rebalancer_dynamic_table [id=caf64b278d8842a39263b02f0a0c8a51]). Expected version 1 got 0
I20240502 08:18:19.362828 1731 tablet_copy_client.cc:783] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:19.363104 1731 tablet_copy_client.cc:647] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:19.363900 1750 tablet.cc:1722] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:19.365522 1746 ts_tablet_manager.cc:902] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1: Initiating tablet copy from peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:19.366393 1731 tablet_copy_client.cc:523] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:19.367837 27116 catalog_manager.cc:5576] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505) does not have the latest schema for tablet ceb53dbc8fd54fa8add21d1bea0c5fc5 (table rebalancer_dynamic_table [id=caf64b278d8842a39263b02f0a0c8a51]). Expected version 1 got 0
I20240502 08:18:19.369938 30289 raft_consensus.cc:1261] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:18:19.370685 27453 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 -> Peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Couldn't send request to peer 1f0bf4bc3c5146539146b825b048bff3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09aa1b5d99114b3095be08c9547fbb54. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:19.370867 1729 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:19.376963 1731 tablet_bootstrap.cc:492] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:19.378756 1729 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:19.383479 30288 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:19.372561 27255 raft_consensus.cc:1261] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:19.394307 1185 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:19.400791 27519 raft_consensus.cc:1203] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:19.404248 1748 tablet.cc:1722] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:19.406075 27115 catalog_manager.cc:5484] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 6 to 7, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:19.406772 1748 tablet.cc:1620] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:19.409878 1749 tablet.cc:1722] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:19.410774 1749 tablet.cc:1620] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:19.414541 27116 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet ceb53dbc8fd54fa8add21d1bea0c5fc5 (table rebalancer_dynamic_table [id=caf64b278d8842a39263b02f0a0c8a51]). Expected version 1 got 0
I20240502 08:18:19.415599 1750 tablet.cc:1620] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: MemRowSet was empty: no flush needed.
I20240502 08:18:19.423972 27115 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=caf64b278d8842a39263b02f0a0c8a51] alter complete (version 1)
I20240502 08:18:19.429241 27255 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:19.453397 1758 ts_tablet_manager.cc:902] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6: Initiating tablet copy from peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:19.457090 1731 tablet_bootstrap.cc:492] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:19.457629 1731 tablet_bootstrap.cc:492] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442: Bootstrap complete.
I20240502 08:18:19.458384 1731 ts_tablet_manager.cc:1367] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.082s user 0.000s sys 0.024s
I20240502 08:18:19.462570 1731 raft_consensus.cc:348] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:19.464618 1731 raft_consensus.cc:729] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: LEARNER
I20240502 08:18:19.465948 1731 consensus_queue.cc:260] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:19.468256 1731 ts_tablet_manager.cc:1398] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 08:18:19.471014 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session bf8ac71b15f740a0a928911360773442-2abc11a238a542e4982452b5ce341101 received from {username='slave'} at 127.20.207.4:32927
I20240502 08:18:19.471119 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session bf8ac71b15f740a0a928911360773442-2abc11a238a542e4982452b5ce341101 on tablet 2abc11a238a542e4982452b5ce341101 with peer bf8ac71b15f740a0a928911360773442
I20240502 08:18:19.479425 1746 tablet_copy_client.cc:308] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.7:40343
I20240502 08:18:19.480211 30310 tablet_copy_service.cc:140] P 49c5c8f70e494edd9fcda40cc0af6cc4: Received BeginTabletCopySession request for tablet 2066c7aa052342a1aca3eb849aa251fc from peer 83e171a5e16c4008923f1d43fc5ce7b1 ({username='slave'} at 127.20.207.2:58817)
I20240502 08:18:19.480432 30310 tablet_copy_service.cc:161] P 49c5c8f70e494edd9fcda40cc0af6cc4: Beginning new tablet copy session on tablet 2066c7aa052342a1aca3eb849aa251fc from peer 83e171a5e16c4008923f1d43fc5ce7b1 at {username='slave'} at 127.20.207.2:58817: session id = 83e171a5e16c4008923f1d43fc5ce7b1-2066c7aa052342a1aca3eb849aa251fc
I20240502 08:18:19.485383 30310 tablet_copy_source_session.cc:205] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:19.486920 1746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2066c7aa052342a1aca3eb849aa251fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.495038 1758 tablet_copy_client.cc:308] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.2:46291
I20240502 08:18:19.495740 27410 tablet_copy_service.cc:140] P 83e171a5e16c4008923f1d43fc5ce7b1: Received BeginTabletCopySession request for tablet 04a0a58eb52f41bf818e5102996d09e0 from peer 84cda20738d2452da166e31fc6da57a6 ({username='slave'} at 127.20.207.6:49833)
I20240502 08:18:19.495913 27410 tablet_copy_service.cc:161] P 83e171a5e16c4008923f1d43fc5ce7b1: Beginning new tablet copy session on tablet 04a0a58eb52f41bf818e5102996d09e0 from peer 84cda20738d2452da166e31fc6da57a6 at {username='slave'} at 127.20.207.6:49833: session id = 84cda20738d2452da166e31fc6da57a6-04a0a58eb52f41bf818e5102996d09e0
I20240502 08:18:19.496587 1746 tablet_copy_client.cc:783] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:19.497215 1746 tablet_copy_client.cc:647] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:19.498287 27410 tablet_copy_source_session.cc:205] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:19.499351 1746 tablet_copy_client.cc:523] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:19.499507 1758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a0a58eb52f41bf818e5102996d09e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.503892 1746 tablet_bootstrap.cc:492] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:19.507265 1758 tablet_copy_client.cc:783] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:19.508761 1758 tablet_copy_client.cc:647] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:19.510293 1758 tablet_copy_client.cc:523] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:19.513602 1758 tablet_bootstrap.cc:492] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:19.521523 1746 tablet_bootstrap.cc:492] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:19.522723 1746 tablet_bootstrap.cc:492] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap complete.
I20240502 08:18:19.523020 1746 ts_tablet_manager.cc:1367] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.004s
I20240502 08:18:19.523804 1746 raft_consensus.cc:348] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:19.524098 1746 raft_consensus.cc:729] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: LEARNER
I20240502 08:18:19.524322 1746 consensus_queue.cc:260] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:19.524957 1746 ts_tablet_manager.cc:1398] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:19.525744 30310 tablet_copy_service.cc:342] P 49c5c8f70e494edd9fcda40cc0af6cc4: Request end of tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-2066c7aa052342a1aca3eb849aa251fc received from {username='slave'} at 127.20.207.2:58817
I20240502 08:18:19.525857 30310 tablet_copy_service.cc:433] P 49c5c8f70e494edd9fcda40cc0af6cc4: ending tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-2066c7aa052342a1aca3eb849aa251fc on tablet 2066c7aa052342a1aca3eb849aa251fc with peer 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:19.531080 30026 consensus_queue.cc:237] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:19.540271 27115 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240502 08:18:19.542038 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 387e9ef5899541b0be27a41dbcf4475c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:19.542179 30156 raft_consensus.cc:1261] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:19.542861 1129 consensus_queue.cc:1035] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:19.543905 27519 raft_consensus.cc:1261] T 387e9ef5899541b0be27a41dbcf4475c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:19.544631 1129 consensus_queue.cc:1035] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:19.549196 1748 tablet.cc:1722] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:19.549652 1748 tablet.cc:1620] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:19.549643 1750 tablet.cc:1722] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:19.550119 1750 tablet.cc:1620] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: MemRowSet was empty: no flush needed.
I20240502 08:18:19.551252 1476 raft_consensus.cc:2932] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:19.553737 1749 tablet.cc:1722] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:19.554241 1749 tablet.cc:1620] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:19.562016 30156 raft_consensus.cc:2932] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:19.570415 27109 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=caf64b278d8842a39263b02f0a0c8a51] alter complete (version 2)
I20240502 08:18:19.582257 27115 catalog_manager.cc:5484] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:19.585856 1758 tablet_bootstrap.cc:492] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:19.586544 1758 tablet_bootstrap.cc:492] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6: Bootstrap complete.
I20240502 08:18:19.586781 1758 ts_tablet_manager.cc:1367] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.073s user 0.054s sys 0.004s
I20240502 08:18:19.587544 1758 raft_consensus.cc:348] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:19.587881 1758 raft_consensus.cc:729] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: LEARNER
I20240502 08:18:19.588150 1758 consensus_queue.cc:260] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:19.588953 1758 ts_tablet_manager.cc:1398] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:19.588917 27516 raft_consensus.cc:2932] T 387e9ef5899541b0be27a41dbcf4475c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:19.593348 27410 tablet_copy_service.cc:342] P 83e171a5e16c4008923f1d43fc5ce7b1: Request end of tablet copy session 84cda20738d2452da166e31fc6da57a6-04a0a58eb52f41bf818e5102996d09e0 received from {username='slave'} at 127.20.207.6:49833
I20240502 08:18:19.593497 27410 tablet_copy_service.cc:433] P 83e171a5e16c4008923f1d43fc5ce7b1: ending tablet copy session 84cda20738d2452da166e31fc6da57a6-04a0a58eb52f41bf818e5102996d09e0 on tablet 04a0a58eb52f41bf818e5102996d09e0 with peer 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:19.611925 27255 consensus_queue.cc:237] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:19.612638 1704 consensus_queue.cc:1035] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.614732 30289 raft_consensus.cc:1261] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:18:19.615391 27191 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Couldn't send request to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15a333c65455473fbdc26d1ec137aebc. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:19.615702 32172 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:19.616883 27648 raft_consensus.cc:1261] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:19.617339 32172 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:19.620451 32172 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:19.625566 27648 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:19.626322 30289 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:19.635344 27115 catalog_manager.cc:5484] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 6 to 7, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:19.639539 27648 raft_consensus.cc:1203] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:19.643957 1541 consensus_queue.cc:1035] T 459e9f5bf6d24ba9b36686d49ff12e2b P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.669100 1796 ts_tablet_manager.cc:902] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:19.670934 1796 tablet_copy_client.cc:308] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:19.671475 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet fef4f56761bb458c8d84807feb38a617 from peer 49c5c8f70e494edd9fcda40cc0af6cc4 ({username='slave'} at 127.20.207.7:58763)
I20240502 08:18:19.671597 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet fef4f56761bb458c8d84807feb38a617 from peer 49c5c8f70e494edd9fcda40cc0af6cc4 at {username='slave'} at 127.20.207.7:58763: session id = 49c5c8f70e494edd9fcda40cc0af6cc4-fef4f56761bb458c8d84807feb38a617
I20240502 08:18:19.674196 27541 tablet_copy_source_session.cc:205] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:19.674811 27110 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:19.678124 1796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef4f56761bb458c8d84807feb38a617. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.684248 1748 tablet.cc:1722] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:19.684235 30156 consensus_queue.cc:237] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:19.684918 1796 tablet_copy_client.cc:783] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:19.685206 1796 tablet_copy_client.cc:647] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:19.686072 1750 tablet.cc:1722] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:19.687974 30026 raft_consensus.cc:1261] T eca2618572fe4900bd420bb59c22381f P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:19.688477 30419 raft_consensus.cc:1261] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:19.688999 1300 consensus_queue.cc:1035] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:19.690101 1569 consensus_queue.cc:1035] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:19.690670 1749 tablet.cc:1722] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240502 08:18:19.692260 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eca2618572fe4900bd420bb59c22381f. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:19.699136 27110 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=caf64b278d8842a39263b02f0a0c8a51] alter complete (version 3)
I20240502 08:18:19.702589 1569 raft_consensus.cc:2932] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:19.707791 30419 raft_consensus.cc:2932] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:19.715198 27116 catalog_manager.cc:5484] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:19.725628 1704 consensus_queue.cc:1035] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.727751 1796 tablet_copy_client.cc:523] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:19.736413 1796 tablet_bootstrap.cc:492] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:19.739264 30026 raft_consensus.cc:2932] T eca2618572fe4900bd420bb59c22381f P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:19.748364 1729 consensus_queue.cc:1035] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.763710 1729 consensus_queue.cc:1035] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.768407 1695 consensus_queue.cc:1035] T bc8fe72edcef41a28fdeea86d501f93b P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.817122 27116 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:19.817330 27116 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:19.826406 27116 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet ceb53dbc8fd54fa8add21d1bea0c5fc5
I20240502 08:18:19.827298 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet ceb53dbc8fd54fa8add21d1bea0c5fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:19 UTC) from {username='slave'} at 127.0.0.1:39166
I20240502 08:18:19.830144 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet ceb53dbc8fd54fa8add21d1bea0c5fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:19 UTC) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:19.830313 30006 tablet_service.cc:1514] Processing DeleteTablet for tablet ceb53dbc8fd54fa8add21d1bea0c5fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:19 UTC) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:19.838621 1818 tablet_replica.cc:330] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:19.838747 27116 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:19.838943 1818 raft_consensus.cc:2229] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:19.840415 1818 raft_consensus.cc:2258] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:19.855474 27629 tablet_service.cc:1467] Processing CreateTablet for tablet 56cb28bc9e264d4882d482499b950349 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=fea00a8313124cad9b42ec6fdac92673]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:19.855885 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cb28bc9e264d4882d482499b950349. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.856213 30400 tablet_service.cc:1467] Processing CreateTablet for tablet 0f26bd4120b64d98bd904a4be5297823 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=fea00a8313124cad9b42ec6fdac92673]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:19.856565 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f26bd4120b64d98bd904a4be5297823. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.856531 27388 raft_consensus.cc:1203] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:19.858440 27237 tablet_service.cc:1467] Processing CreateTablet for tablet 0f26bd4120b64d98bd904a4be5297823 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=fea00a8313124cad9b42ec6fdac92673]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:19.858867 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f26bd4120b64d98bd904a4be5297823. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.860249 27366 tablet_service.cc:1467] Processing CreateTablet for tablet 556fc9ebf7bd4b859f584d46411c2403 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=fea00a8313124cad9b42ec6fdac92673]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:19.860638 27366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 556fc9ebf7bd4b859f584d46411c2403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.863054 1818 ts_tablet_manager.cc:1875] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:19.863355 30138 tablet_service.cc:1467] Processing CreateTablet for tablet 56cb28bc9e264d4882d482499b950349 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=fea00a8313124cad9b42ec6fdac92673]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:19.863842 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cb28bc9e264d4882d482499b950349. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.865523 27498 tablet_service.cc:1467] Processing CreateTablet for tablet 56cb28bc9e264d4882d482499b950349 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=fea00a8313124cad9b42ec6fdac92673]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:19.865886 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cb28bc9e264d4882d482499b950349. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.863351 30139 tablet_service.cc:1467] Processing CreateTablet for tablet 0f26bd4120b64d98bd904a4be5297823 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=fea00a8313124cad9b42ec6fdac92673]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:19.866816 30006 tablet_service.cc:1467] Processing CreateTablet for tablet 556fc9ebf7bd4b859f584d46411c2403 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=fea00a8313124cad9b42ec6fdac92673]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:19.867053 30139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f26bd4120b64d98bd904a4be5297823. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.867007 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 556fc9ebf7bd4b859f584d46411c2403 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=fea00a8313124cad9b42ec6fdac92673]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:19.867198 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 556fc9ebf7bd4b859f584d46411c2403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.867367 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 556fc9ebf7bd4b859f584d46411c2403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.875829 1712 raft_consensus.cc:1050] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3: attempting to promote NON_VOTER b01e7c8a302d41078fa39cfef665ff59 to VOTER
I20240502 08:18:19.876595 1712 consensus_queue.cc:237] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:19.891773 1818 ts_tablet_manager.cc:1888] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:19.896530 1823 tablet_bootstrap.cc:492] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:19.897778 1817 tablet_replica.cc:330] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:19.898108 1817 raft_consensus.cc:2229] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:19.898272 27516 raft_consensus.cc:1261] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:19.898422 1817 raft_consensus.cc:2258] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:19.898840 27648 raft_consensus.cc:1261] T d00f25f29f5d4db2a2261dd37c0793fa P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:19.899364 1817 ts_tablet_manager.cc:1875] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:19.899751 1825 tablet_bootstrap.cc:492] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:19.905248 1823 tablet_bootstrap.cc:654] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.907181 1816 tablet_replica.cc:330] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:18:19.907325 1825 tablet_bootstrap.cc:654] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.907469 1816 raft_consensus.cc:2229] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:19.907732 1816 raft_consensus.cc:2258] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:19.908799 1816 ts_tablet_manager.cc:1875] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:19.893473 1820 ts_tablet_manager.cc:902] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:19.898947 1129 consensus_queue.cc:1035] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:19.904670 1829 tablet_bootstrap.cc:492] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:19.914458 30156 raft_consensus.cc:1261] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:19.915195 1129 consensus_queue.cc:1035] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:19.916538 1829 tablet_bootstrap.cc:654] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.918735 1818 log.cc:1196] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/ceb53dbc8fd54fa8add21d1bea0c5fc5
I20240502 08:18:19.919306 1818 ts_tablet_manager.cc:1909] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1f0bf4bc3c5146539146b825b048bff3: Deleting consensus metadata
I20240502 08:18:19.920728 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet ceb53dbc8fd54fa8add21d1bea0c5fc5 (table rebalancer_dynamic_table_new_name [id=caf64b278d8842a39263b02f0a0c8a51]) successfully deleted
I20240502 08:18:19.920948 1712 consensus_queue.cc:1035] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:19.931255 27110 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:19.937492 1476 raft_consensus.cc:2932] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:19.942210 27516 raft_consensus.cc:2932] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:19.942639 1829 tablet_bootstrap.cc:492] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:19.945228 1829 ts_tablet_manager.cc:1367] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.001s
I20240502 08:18:19.945684 30157 raft_consensus.cc:2932] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:19.947052 1829 raft_consensus.cc:348] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:19.946171 1825 tablet_bootstrap.cc:492] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:19.947536 1825 ts_tablet_manager.cc:1367] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.000s
I20240502 08:18:19.948568 27110 catalog_manager.cc:5484] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:19.950632 1825 raft_consensus.cc:348] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:19.953358 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 50cdc758e790419e91c4ebacf9307c85 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bdd7dc8b8bdf4373b3c4d2c3e428e2ae]), partition=
I20240502 08:18:19.953878 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50cdc758e790419e91c4ebacf9307c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.954061 1829 raft_consensus.cc:374] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.954150 1829 raft_consensus.cc:729] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.954351 1829 consensus_queue.cc:260] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:19.954983 1829 ts_tablet_manager.cc:1398] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240502 08:18:19.955088 30006 tablet_service.cc:1467] Processing CreateTablet for tablet 50cdc758e790419e91c4ebacf9307c85 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bdd7dc8b8bdf4373b3c4d2c3e428e2ae]), partition=
I20240502 08:18:19.955179 1825 raft_consensus.cc:374] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.955276 1825 raft_consensus.cc:729] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.955394 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50cdc758e790419e91c4ebacf9307c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.955647 1825 consensus_queue.cc:260] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:19.956188 1825 ts_tablet_manager.cc:1398] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 08:18:19.957765 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 50cdc758e790419e91c4ebacf9307c85 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bdd7dc8b8bdf4373b3c4d2c3e428e2ae]), partition=
I20240502 08:18:19.958153 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50cdc758e790419e91c4ebacf9307c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.964982 1816 ts_tablet_manager.cc:1888] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:19.966517 1816 log.cc:1196] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/ceb53dbc8fd54fa8add21d1bea0c5fc5
I20240502 08:18:19.966296 27648 raft_consensus.cc:2932] T d00f25f29f5d4db2a2261dd37c0793fa P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:19.967326 1816 ts_tablet_manager.cc:1909] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting consensus metadata
I20240502 08:18:19.969472 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet ceb53dbc8fd54fa8add21d1bea0c5fc5 (table rebalancer_dynamic_table_new_name [id=caf64b278d8842a39263b02f0a0c8a51]) successfully deleted
I20240502 08:18:19.975214 1826 tablet_bootstrap.cc:492] T 556fc9ebf7bd4b859f584d46411c2403 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:19.976895 1826 tablet_bootstrap.cc:654] T 556fc9ebf7bd4b859f584d46411c2403 P 83e171a5e16c4008923f1d43fc5ce7b1: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.978088 1821 tablet_bootstrap.cc:492] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:19.981899 1817 ts_tablet_manager.cc:1888] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:19.982383 1821 tablet_bootstrap.cc:654] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.982800 1817 log.cc:1196] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/ceb53dbc8fd54fa8add21d1bea0c5fc5
I20240502 08:18:19.983369 1817 ts_tablet_manager.cc:1909] T ceb53dbc8fd54fa8add21d1bea0c5fc5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting consensus metadata
I20240502 08:18:19.984329 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet ceb53dbc8fd54fa8add21d1bea0c5fc5 (table rebalancer_dynamic_table_new_name [id=caf64b278d8842a39263b02f0a0c8a51]) successfully deleted
I20240502 08:18:19.984545 1822 tablet_bootstrap.cc:492] T 0f26bd4120b64d98bd904a4be5297823 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:19.987963 1827 tablet_bootstrap.cc:492] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:19.992336 1822 tablet_bootstrap.cc:654] T 0f26bd4120b64d98bd904a4be5297823 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.990592 1828 tablet_bootstrap.cc:492] T 0f26bd4120b64d98bd904a4be5297823 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:19.996260 1828 tablet_bootstrap.cc:654] T 0f26bd4120b64d98bd904a4be5297823 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.996824 1827 tablet_bootstrap.cc:654] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:20.000824 30025 consensus_queue.cc:237] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:20.003568 27516 raft_consensus.cc:1261] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:20.004303 1476 consensus_queue.cc:1035] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:20.006486 30156 raft_consensus.cc:1203] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:18:20.007014 1823 tablet_bootstrap.cc:492] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:20.006743 1129 raft_consensus.cc:2932] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:20.007247 1823 ts_tablet_manager.cc:1367] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.111s user 0.005s sys 0.000s
I20240502 08:18:20.008236 1823 raft_consensus.cc:348] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:20.008527 1823 raft_consensus.cc:374] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.011039 1829 tablet_bootstrap.cc:492] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:20.011682 27516 raft_consensus.cc:2932] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:20.013127 1823 raft_consensus.cc:729] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.013438 1823 consensus_queue.cc:260] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:20.015561 1823 ts_tablet_manager.cc:1398] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:18:20.016696 1823 tablet_bootstrap.cc:492] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:20.021416 1828 tablet_bootstrap.cc:492] T 0f26bd4120b64d98bd904a4be5297823 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:20.021565 1828 ts_tablet_manager.cc:1367] T 0f26bd4120b64d98bd904a4be5297823 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20240502 08:18:20.022430 1828 raft_consensus.cc:348] T 0f26bd4120b64d98bd904a4be5297823 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:20.022630 1828 raft_consensus.cc:374] T 0f26bd4120b64d98bd904a4be5297823 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.022718 1828 raft_consensus.cc:729] T 0f26bd4120b64d98bd904a4be5297823 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.022933 1828 consensus_queue.cc:260] T 0f26bd4120b64d98bd904a4be5297823 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:20.032554 27113 catalog_manager.cc:5484] T d00f25f29f5d4db2a2261dd37c0793fa P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:20.034538 1828 ts_tablet_manager.cc:1398] T 0f26bd4120b64d98bd904a4be5297823 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 08:18:20.034178 1829 tablet_bootstrap.cc:654] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:20.036134 1845 ts_tablet_manager.cc:902] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:20.037101 1828 tablet_bootstrap.cc:492] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:20.038659 30139 tablet_service.cc:1514] Processing DeleteTablet for tablet d00f25f29f5d4db2a2261dd37c0793fa with delete_type TABLET_DATA_TOMBSTONED (TS 84cda20738d2452da166e31fc6da57a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:20.039170 30025 consensus_queue.cc:237] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:20.041615 1826 tablet_bootstrap.cc:492] T 556fc9ebf7bd4b859f584d46411c2403 P 83e171a5e16c4008923f1d43fc5ce7b1: No bootstrap required, opened a new log
I20240502 08:18:20.041978 1823 tablet_bootstrap.cc:654] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:20.042704 1821 tablet_bootstrap.cc:492] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:20.042886 1850 tablet_replica.cc:330] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:20.043532 1826 ts_tablet_manager.cc:1367] T 556fc9ebf7bd4b859f584d46411c2403 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.000s
I20240502 08:18:20.045100 1850 raft_consensus.cc:2229] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:20.045214 1821 ts_tablet_manager.cc:1367] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.000s
I20240502 08:18:20.046422 1821 raft_consensus.cc:348] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:20.047433 1826 raft_consensus.cc:348] T 556fc9ebf7bd4b859f584d46411c2403 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:20.049754 30423 raft_consensus.cc:1261] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:20.050222 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d00f25f29f5d4db2a2261dd37c0793fa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:20.050340 1712 consensus_queue.cc:1035] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:20.050473 1850 raft_consensus.cc:2258] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:18:20.052726 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70f4c40bc22947468dfd15804a4e66cd. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:20.053680 27516 raft_consensus.cc:1261] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:20.054401 1827 tablet_bootstrap.cc:492] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:20.054380 1831 consensus_queue.cc:1035] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:20.055887 1822 tablet_bootstrap.cc:492] T 0f26bd4120b64d98bd904a4be5297823 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:20.056128 1826 raft_consensus.cc:374] T 556fc9ebf7bd4b859f584d46411c2403 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.056340 1826 raft_consensus.cc:729] T 556fc9ebf7bd4b859f584d46411c2403 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.056560 1827 ts_tablet_manager.cc:1367] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.069s user 0.006s sys 0.000s
I20240502 08:18:20.056913 1826 consensus_queue.cc:260] T 556fc9ebf7bd4b859f584d46411c2403 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:20.052922 27650 raft_consensus.cc:1261] T d00f25f29f5d4db2a2261dd37c0793fa P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:20.057982 1129 consensus_queue.cc:1035] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:20.058218 1821 raft_consensus.cc:374] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.060034 1821 raft_consensus.cc:729] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.060252 1822 ts_tablet_manager.cc:1367] T 0f26bd4120b64d98bd904a4be5297823 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.076s user 0.007s sys 0.000s
I20240502 08:18:20.061880 1822 raft_consensus.cc:348] T 0f26bd4120b64d98bd904a4be5297823 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:20.062340 1821 consensus_queue.cc:260] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:20.064692 1826 ts_tablet_manager.cc:1398] T 556fc9ebf7bd4b859f584d46411c2403 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240502 08:18:20.066867 1828 tablet_bootstrap.cc:654] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:20.067764 1823 tablet_bootstrap.cc:492] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:20.067981 1823 ts_tablet_manager.cc:1367] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.004s
I20240502 08:18:20.068497 1827 raft_consensus.cc:348] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:20.068783 1827 raft_consensus.cc:374] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.068889 1827 raft_consensus.cc:729] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.068831 1823 raft_consensus.cc:348] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:20.069113 1823 raft_consensus.cc:374] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.069223 1823 raft_consensus.cc:729] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.069205 1827 consensus_queue.cc:260] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:20.069569 1823 consensus_queue.cc:260] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:20.071452 1823 ts_tablet_manager.cc:1398] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:18:20.072192 1827 ts_tablet_manager.cc:1398] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 08:18:20.074167 1821 ts_tablet_manager.cc:1398] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240502 08:18:20.074501 1821 tablet_bootstrap.cc:492] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:20.076118 1821 tablet_bootstrap.cc:654] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:20.079852 1831 raft_consensus.cc:2932] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:20.064936 27648 raft_consensus.cc:2932] T d00f25f29f5d4db2a2261dd37c0793fa P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:20.086710 30418 raft_consensus.cc:2932] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:20.088896 1822 raft_consensus.cc:374] T 0f26bd4120b64d98bd904a4be5297823 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.088970 1822 raft_consensus.cc:729] T 0f26bd4120b64d98bd904a4be5297823 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.089192 1822 consensus_queue.cc:260] T 0f26bd4120b64d98bd904a4be5297823 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:20.090725 1822 ts_tablet_manager.cc:1398] T 0f26bd4120b64d98bd904a4be5297823 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20240502 08:18:20.096758 27114 catalog_manager.cc:5484] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:20.102604 1829 tablet_bootstrap.cc:492] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:20.106587 1860 ts_tablet_manager.cc:902] T eca2618572fe4900bd420bb59c22381f P 49c5c8f70e494edd9fcda40cc0af6cc4: Initiating tablet copy from peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:20.108872 1829 ts_tablet_manager.cc:1367] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.098s user 0.000s sys 0.006s
I20240502 08:18:20.109931 1821 tablet_bootstrap.cc:492] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:20.110110 1821 ts_tablet_manager.cc:1367] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20240502 08:18:20.111088 1829 raft_consensus.cc:348] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:20.111335 1829 raft_consensus.cc:374] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.111426 1829 raft_consensus.cc:729] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.112193 1821 raft_consensus.cc:348] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:20.112639 1821 raft_consensus.cc:374] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.112701 1821 raft_consensus.cc:729] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.111696 1829 consensus_queue.cc:260] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:20.115382 1185 raft_consensus.cc:482] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:20.115531 1185 raft_consensus.cc:504] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:20.116142 1185 leader_election.cc:288] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:20.117079 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cb28bc9e264d4882d482499b950349" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
W20240502 08:18:20.118130 27456 leader_election.cc:341] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:20.117945 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cb28bc9e264d4882d482499b950349" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:20.121052 27650 raft_consensus.cc:2454] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 0.
I20240502 08:18:20.121466 27456 leader_election.cc:302] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters: 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:20.113914 1821 consensus_queue.cc:260] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:20.122220 1729 raft_consensus.cc:2781] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:20.122299 1729 raft_consensus.cc:482] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:20.122373 1729 raft_consensus.cc:3037] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.123778 1856 ts_tablet_manager.cc:902] T 387e9ef5899541b0be27a41dbcf4475c P 83e171a5e16c4008923f1d43fc5ce7b1: Initiating tablet copy from peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:20.125088 1729 raft_consensus.cc:504] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:20.126065 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cb28bc9e264d4882d482499b950349" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:20.125890 27516 raft_consensus.cc:2932] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:20.126245 27650 raft_consensus.cc:3037] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.126296 1828 tablet_bootstrap.cc:492] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:20.126487 1828 ts_tablet_manager.cc:1367] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.089s user 0.004s sys 0.000s
I20240502 08:18:20.128719 27650 raft_consensus.cc:2454] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:18:20.132270 1828 raft_consensus.cc:348] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:20.132498 1828 raft_consensus.cc:374] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.132591 1828 raft_consensus.cc:729] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.132709 27456 leader_election.cc:302] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:20.132966 1820 tablet_copy_client.cc:308] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:20.132860 1828 consensus_queue.cc:260] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:20.133118 1185 raft_consensus.cc:2781] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:20.133245 1185 raft_consensus.cc:686] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Leader. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:18:20.133471 1185 consensus_queue.cc:237] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:20.134104 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet 09aa1b5d99114b3095be08c9547fbb54 from peer 1f0bf4bc3c5146539146b825b048bff3 ({username='slave'} at 127.20.207.5:54501)
I20240502 08:18:20.134236 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet 09aa1b5d99114b3095be08c9547fbb54 from peer 1f0bf4bc3c5146539146b825b048bff3 at {username='slave'} at 127.20.207.5:54501: session id = 1f0bf4bc3c5146539146b825b048bff3-09aa1b5d99114b3095be08c9547fbb54
I20240502 08:18:20.135087 1850 ts_tablet_manager.cc:1875] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:20.136993 27541 tablet_copy_source_session.cc:205] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:20.133606 1828 ts_tablet_manager.cc:1398] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 08:18:20.141091 1821 ts_tablet_manager.cc:1398] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20240502 08:18:20.142036 1829 ts_tablet_manager.cc:1398] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:18:20.144071 1729 leader_election.cc:288] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:20.146663 1845 tablet_copy_client.cc:308] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:18:20.147461 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cb28bc9e264d4882d482499b950349" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:20.147624 30157 raft_consensus.cc:3037] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.147891 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet 15a333c65455473fbdc26d1ec137aebc from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad ({username='slave'} at 127.20.207.8:40413)
I20240502 08:18:20.148047 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet 15a333c65455473fbdc26d1ec137aebc from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad at {username='slave'} at 127.20.207.8:40413: session id = 31a1e33dcdbf42f58ea72ce1c0afb0ad-15a333c65455473fbdc26d1ec137aebc
I20240502 08:18:20.151196 27278 tablet_copy_source_session.cc:205] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:20.152693 30157 raft_consensus.cc:2454] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01e7c8a302d41078fa39cfef665ff59 in term 1.
I20240502 08:18:20.152841 1845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a333c65455473fbdc26d1ec137aebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.158528 1845 tablet_copy_client.cc:783] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:20.158840 1845 tablet_copy_client.cc:647] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:20.160370 1845 tablet_copy_client.cc:523] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:20.161985 1850 ts_tablet_manager.cc:1888] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:20.162292 1850 log.cc:1196] T d00f25f29f5d4db2a2261dd37c0793fa P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/d00f25f29f5d4db2a2261dd37c0793fa
I20240502 08:18:20.163103 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet d00f25f29f5d4db2a2261dd37c0793fa (table rebalancer_extra_table_12 [id=e2a08da4858541088d7e3e5a7ae41e45]) successfully deleted
I20240502 08:18:20.163677 1845 tablet_bootstrap.cc:492] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:20.166536 1820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09aa1b5d99114b3095be08c9547fbb54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.189870 1796 tablet_bootstrap.cc:492] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:20.202498 1729 raft_consensus.cc:1050] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER bf8ac71b15f740a0a928911360773442 to VOTER
I20240502 08:18:20.203433 1729 consensus_queue.cc:237] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:20.204756 1796 tablet_bootstrap.cc:492] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap complete.
I20240502 08:18:20.206869 1796 ts_tablet_manager.cc:1367] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.471s user 0.046s sys 0.009s
I20240502 08:18:20.210793 1796 raft_consensus.cc:348] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:20.211511 1502 raft_consensus.cc:482] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:20.211659 1502 raft_consensus.cc:504] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:20.212320 1502 leader_election.cc:288] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:20.212558 1796 raft_consensus.cc:729] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: LEARNER
I20240502 08:18:20.213004 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50cdc758e790419e91c4ebacf9307c85" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:20.213241 30289 raft_consensus.cc:2454] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:18:20.213225 27110 catalog_manager.cc:5484] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3). New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:20.213670 27584 leader_election.cc:302] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:20.214291 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50cdc758e790419e91c4ebacf9307c85" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:20.214531 30025 raft_consensus.cc:2454] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:18:20.214787 1502 raft_consensus.cc:2781] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:20.214877 1502 raft_consensus.cc:482] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:20.214946 1502 raft_consensus.cc:3037] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.217469 1502 raft_consensus.cc:504] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:20.221256 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50cdc758e790419e91c4ebacf9307c85" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:20.221534 30025 raft_consensus.cc:3037] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.223598 27650 raft_consensus.cc:1261] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:20.224233 30025 raft_consensus.cc:2454] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:20.225862 1729 consensus_queue.cc:1035] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:20.229364 30025 raft_consensus.cc:1261] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:20.229614 30418 raft_consensus.cc:1261] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:20.229905 27584 leader_election.cc:302] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:20.230309 1311 raft_consensus.cc:2781] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:20.233923 1502 leader_election.cc:288] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:20.234596 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50cdc758e790419e91c4ebacf9307c85" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:20.234880 30289 raft_consensus.cc:3037] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.235345 1820 tablet_copy_client.cc:783] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:20.235658 1820 tablet_copy_client.cc:647] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:20.239886 30289 raft_consensus.cc:2454] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:20.241721 1856 tablet_copy_client.cc:308] T 387e9ef5899541b0be27a41dbcf4475c P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.5:37815
I20240502 08:18:20.242589 30048 tablet_copy_service.cc:140] P 1f0bf4bc3c5146539146b825b048bff3: Received BeginTabletCopySession request for tablet 387e9ef5899541b0be27a41dbcf4475c from peer 83e171a5e16c4008923f1d43fc5ce7b1 ({username='slave'} at 127.20.207.2:49697)
I20240502 08:18:20.242727 30048 tablet_copy_service.cc:161] P 1f0bf4bc3c5146539146b825b048bff3: Beginning new tablet copy session on tablet 387e9ef5899541b0be27a41dbcf4475c from peer 83e171a5e16c4008923f1d43fc5ce7b1 at {username='slave'} at 127.20.207.2:49697: session id = 83e171a5e16c4008923f1d43fc5ce7b1-387e9ef5899541b0be27a41dbcf4475c
I20240502 08:18:20.245838 30048 tablet_copy_source_session.cc:205] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:20.249697 1185 consensus_queue.cc:1035] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:20.252204 1856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387e9ef5899541b0be27a41dbcf4475c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.252429 1185 consensus_queue.cc:1035] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:20.266283 1856 tablet_copy_client.cc:783] T 387e9ef5899541b0be27a41dbcf4475c P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:20.266510 1856 tablet_copy_client.cc:647] T 387e9ef5899541b0be27a41dbcf4475c P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:20.267964 1856 tablet_copy_client.cc:523] T 387e9ef5899541b0be27a41dbcf4475c P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:20.269553 1820 tablet_copy_client.cc:523] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:20.271111 1311 raft_consensus.cc:686] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:18:20.271445 1311 consensus_queue.cc:237] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } }
I20240502 08:18:20.271770 1856 tablet_bootstrap.cc:492] T 387e9ef5899541b0be27a41dbcf4475c P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:20.213012 1796 consensus_queue.cc:260] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:20.278756 1796 ts_tablet_manager.cc:1398] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.069s user 0.002s sys 0.000s
I20240502 08:18:20.280820 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-fef4f56761bb458c8d84807feb38a617 received from {username='slave'} at 127.20.207.7:58763
I20240502 08:18:20.280939 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-fef4f56761bb458c8d84807feb38a617 on tablet fef4f56761bb458c8d84807feb38a617 with peer 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:20.284592 1729 raft_consensus.cc:2932] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:20.286680 27110 catalog_manager.cc:5484] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:20.289224 1820 tablet_bootstrap.cc:492] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:20.295476 623 raft_consensus.cc:482] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:20.295626 623 raft_consensus.cc:504] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:20.296248 623 leader_election.cc:288] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:20.296881 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556fc9ebf7bd4b859f584d46411c2403" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:20.297116 30025 raft_consensus.cc:2454] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:20.297526 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556fc9ebf7bd4b859f584d46411c2403" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:18:20.297744 27388 raft_consensus.cc:2454] T 556fc9ebf7bd4b859f584d46411c2403 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:20.298424 30225 leader_election.cc:302] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:20.298977 623 raft_consensus.cc:2781] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:20.299069 623 raft_consensus.cc:482] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:20.299161 623 raft_consensus.cc:3037] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.299804 1856 tablet_bootstrap.cc:492] T 387e9ef5899541b0be27a41dbcf4475c P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:20.300168 1856 tablet_bootstrap.cc:492] T 387e9ef5899541b0be27a41dbcf4475c P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap complete.
I20240502 08:18:20.300403 1856 ts_tablet_manager.cc:1367] T 387e9ef5899541b0be27a41dbcf4475c P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.002s
I20240502 08:18:20.301126 1856 raft_consensus.cc:348] T 387e9ef5899541b0be27a41dbcf4475c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:20.301334 1856 raft_consensus.cc:729] T 387e9ef5899541b0be27a41dbcf4475c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: LEARNER
I20240502 08:18:20.301517 1856 consensus_queue.cc:260] T 387e9ef5899541b0be27a41dbcf4475c P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:20.302095 1856 ts_tablet_manager.cc:1398] T 387e9ef5899541b0be27a41dbcf4475c P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 08:18:20.302744 623 raft_consensus.cc:504] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:20.304065 27388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556fc9ebf7bd4b859f584d46411c2403" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:18:20.304219 27388 raft_consensus.cc:3037] T 556fc9ebf7bd4b859f584d46411c2403 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.307221 27388 raft_consensus.cc:2454] T 556fc9ebf7bd4b859f584d46411c2403 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:20.307643 30048 tablet_copy_service.cc:342] P 1f0bf4bc3c5146539146b825b048bff3: Request end of tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-387e9ef5899541b0be27a41dbcf4475c received from {username='slave'} at 127.20.207.2:49697
I20240502 08:18:20.307770 30048 tablet_copy_service.cc:433] P 1f0bf4bc3c5146539146b825b048bff3: ending tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-387e9ef5899541b0be27a41dbcf4475c on tablet 387e9ef5899541b0be27a41dbcf4475c with peer 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:20.308368 30225 leader_election.cc:302] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:20.308709 1661 raft_consensus.cc:2781] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:20.312230 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556fc9ebf7bd4b859f584d46411c2403" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:20.312506 30025 raft_consensus.cc:3037] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.312350 623 leader_election.cc:288] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:20.315240 30025 raft_consensus.cc:2454] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:20.318773 1661 raft_consensus.cc:686] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:18:20.319149 1661 consensus_queue.cc:237] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:20.325655 1860 tablet_copy_client.cc:308] T eca2618572fe4900bd420bb59c22381f P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.6:38899
I20240502 08:18:20.326376 30179 tablet_copy_service.cc:140] P 84cda20738d2452da166e31fc6da57a6: Received BeginTabletCopySession request for tablet eca2618572fe4900bd420bb59c22381f from peer 49c5c8f70e494edd9fcda40cc0af6cc4 ({username='slave'} at 127.20.207.7:60051)
I20240502 08:18:20.326530 30179 tablet_copy_service.cc:161] P 84cda20738d2452da166e31fc6da57a6: Beginning new tablet copy session on tablet eca2618572fe4900bd420bb59c22381f from peer 49c5c8f70e494edd9fcda40cc0af6cc4 at {username='slave'} at 127.20.207.7:60051: session id = 49c5c8f70e494edd9fcda40cc0af6cc4-eca2618572fe4900bd420bb59c22381f
I20240502 08:18:20.329202 30179 tablet_copy_source_session.cc:205] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:20.330833 1860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca2618572fe4900bd420bb59c22381f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.331072 27650 raft_consensus.cc:2932] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:20.333519 27109 catalog_manager.cc:5484] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7). New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:20.336993 30025 raft_consensus.cc:2932] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:20.338923 30418 raft_consensus.cc:2932] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:20.341056 1860 tablet_copy_client.cc:783] T eca2618572fe4900bd420bb59c22381f P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:20.341282 1860 tablet_copy_client.cc:647] T eca2618572fe4900bd420bb59c22381f P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:20.347050 27109 catalog_manager.cc:5484] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:20.355633 27519 consensus_queue.cc:237] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:20.357718 1860 tablet_copy_client.cc:523] T eca2618572fe4900bd420bb59c22381f P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:20.362075 27648 raft_consensus.cc:1261] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:20.364161 1860 tablet_bootstrap.cc:492] T eca2618572fe4900bd420bb59c22381f P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:20.364035 1891 consensus_queue.cc:1035] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:20.366238 1866 raft_consensus.cc:2932] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:20.373529 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2abc11a238a542e4982452b5ce341101 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:20.376694 27650 raft_consensus.cc:2932] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:20.380633 1845 tablet_bootstrap.cc:492] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:20.383389 27109 catalog_manager.cc:5484] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:20.384908 1845 tablet_bootstrap.cc:492] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap complete.
I20240502 08:18:20.385186 1845 ts_tablet_manager.cc:1367] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.222s user 0.055s sys 0.007s
I20240502 08:18:20.385910 1888 raft_consensus.cc:1050] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4: attempting to promote NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 to VOTER
I20240502 08:18:20.385850 1845 raft_consensus.cc:348] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:20.386106 1845 raft_consensus.cc:729] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: LEARNER
I20240502 08:18:20.386288 1845 consensus_queue.cc:260] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:20.386976 1888 consensus_queue.cc:237] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:20.390262 30418 raft_consensus.cc:1261] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:20.390837 1661 consensus_queue.cc:1035] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:20.391093 27255 raft_consensus.cc:1261] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:20.393853 1661 consensus_queue.cc:1035] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:20.394308 27388 raft_consensus.cc:1261] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:20.396036 623 consensus_queue.cc:1035] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:20.404346 30006 tablet_service.cc:1514] Processing DeleteTablet for tablet 2abc11a238a542e4982452b5ce341101 with delete_type TABLET_DATA_TOMBSTONED (TS 1f0bf4bc3c5146539146b825b048bff3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:20.404659 1818 tablet_replica.cc:330] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:20.404831 1818 raft_consensus.cc:2229] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:20.404999 1818 raft_consensus.cc:2258] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:20.407527 1747 raft_consensus.cc:482] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:20.407653 1747 raft_consensus.cc:504] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:20.408356 1747 leader_election.cc:288] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:20.408633 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f26bd4120b64d98bd904a4be5297823" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:20.408885 30157 raft_consensus.cc:2454] T 0f26bd4120b64d98bd904a4be5297823 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:20.408870 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f26bd4120b64d98bd904a4be5297823" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:20.409078 27255 raft_consensus.cc:2454] T 0f26bd4120b64d98bd904a4be5297823 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 0.
I20240502 08:18:20.409413 1818 ts_tablet_manager.cc:1875] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:20.409464 30357 leader_election.cc:302] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters:
I20240502 08:18:20.409739 1747 raft_consensus.cc:2781] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:20.409848 1747 raft_consensus.cc:482] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:20.409941 1747 raft_consensus.cc:3037] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.412698 1747 raft_consensus.cc:504] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:20.413678 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f26bd4120b64d98bd904a4be5297823" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:20.413765 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f26bd4120b64d98bd904a4be5297823" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:20.413862 30157 raft_consensus.cc:3037] T 0f26bd4120b64d98bd904a4be5297823 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.413918 27257 raft_consensus.cc:3037] T 0f26bd4120b64d98bd904a4be5297823 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.416499 27257 raft_consensus.cc:2454] T 0f26bd4120b64d98bd904a4be5297823 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:20.416591 1818 ts_tablet_manager.cc:1888] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:20.416712 1818 log.cc:1196] T 2abc11a238a542e4982452b5ce341101 P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/2abc11a238a542e4982452b5ce341101
I20240502 08:18:20.417069 30357 leader_election.cc:302] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters:
I20240502 08:18:20.417320 1747 leader_election.cc:288] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 1 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:20.417344 1755 raft_consensus.cc:2781] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:20.417452 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet 2abc11a238a542e4982452b5ce341101 (table rebalancer_extra_table_18 [id=2c7cf528be404fbca449af1fcf7d0e23]) successfully deleted
I20240502 08:18:20.417513 1755 raft_consensus.cc:686] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:18:20.417851 1755 consensus_queue.cc:237] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:20.418797 1845 ts_tablet_manager.cc:1398] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20240502 08:18:20.419358 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-15a333c65455473fbdc26d1ec137aebc received from {username='slave'} at 127.20.207.8:40413
I20240502 08:18:20.419453 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-15a333c65455473fbdc26d1ec137aebc on tablet 15a333c65455473fbdc26d1ec137aebc with peer 31a1e33dcdbf42f58ea72ce1c0afb0ad
I20240502 08:18:20.420090 30418 raft_consensus.cc:1261] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:20.420684 1866 consensus_queue.cc:1035] T 2abc11a238a542e4982452b5ce341101 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:20.421085 30157 raft_consensus.cc:2454] T 0f26bd4120b64d98bd904a4be5297823 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:20.423187 30418 raft_consensus.cc:2932] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:20.424443 1860 tablet_bootstrap.cc:492] T eca2618572fe4900bd420bb59c22381f P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:20.425388 1860 tablet_bootstrap.cc:492] T eca2618572fe4900bd420bb59c22381f P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap complete.
I20240502 08:18:20.425665 1860 ts_tablet_manager.cc:1367] T eca2618572fe4900bd420bb59c22381f P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.007s
I20240502 08:18:20.426546 1860 raft_consensus.cc:348] T eca2618572fe4900bd420bb59c22381f P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:20.426939 1860 raft_consensus.cc:729] T eca2618572fe4900bd420bb59c22381f P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: LEARNER
I20240502 08:18:20.427418 27109 catalog_manager.cc:5484] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:20.427232 1860 consensus_queue.cc:260] T eca2618572fe4900bd420bb59c22381f P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:20.429572 1860 ts_tablet_manager.cc:1398] T eca2618572fe4900bd420bb59c22381f P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:20.431648 30179 tablet_copy_service.cc:342] P 84cda20738d2452da166e31fc6da57a6: Request end of tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-eca2618572fe4900bd420bb59c22381f received from {username='slave'} at 127.20.207.7:60051
I20240502 08:18:20.431774 30179 tablet_copy_service.cc:433] P 84cda20738d2452da166e31fc6da57a6: ending tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-eca2618572fe4900bd420bb59c22381f on tablet eca2618572fe4900bd420bb59c22381f with peer 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:20.443156 30289 consensus_queue.cc:237] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:20.450103 30025 raft_consensus.cc:1261] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:20.451051 1888 consensus_queue.cc:1035] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:20.451653 30226 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 -> Peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Couldn't send request to peer 84cda20738d2452da166e31fc6da57a6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65ac899c820a43e4ab57454e224edc83. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:20.452219 1888 raft_consensus.cc:2932] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:20.452787 27650 raft_consensus.cc:1261] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:20.455361 1661 consensus_queue.cc:1035] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:20.456923 27390 raft_consensus.cc:2932] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:20.462527 27109 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:20.465605 27114 catalog_manager.cc:5484] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:20.466975 27255 raft_consensus.cc:2932] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:20.468016 30419 raft_consensus.cc:2932] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:20.482581 1894 raft_consensus.cc:2932] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:20.489323 27648 raft_consensus.cc:2932] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:20.498059 27115 catalog_manager.cc:5484] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index -1 to 2, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:20.506312 30028 raft_consensus.cc:2932] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:20.507254 30289 consensus_queue.cc:237] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:20.509172 27390 raft_consensus.cc:1261] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:20.512988 27255 raft_consensus.cc:1261] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:20.513604 1661 consensus_queue.cc:1035] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:20.514284 1661 consensus_queue.cc:1035] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:20.519008 1661 raft_consensus.cc:2932] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:20.522560 27255 raft_consensus.cc:2932] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:20.522979 27388 raft_consensus.cc:1203] T 387e9ef5899541b0be27a41dbcf4475c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:20.523264 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2066c7aa052342a1aca3eb849aa251fc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:20.525745 27389 raft_consensus.cc:2932] T 2066c7aa052342a1aca3eb849aa251fc P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:20.526513 27114 catalog_manager.cc:5484] T 2066c7aa052342a1aca3eb849aa251fc P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:20.535595 30400 tablet_service.cc:1514] Processing DeleteTablet for tablet 2066c7aa052342a1aca3eb849aa251fc with delete_type TABLET_DATA_TOMBSTONED (TS 31a1e33dcdbf42f58ea72ce1c0afb0ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:20.542049 1820 tablet_bootstrap.cc:492] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:20.542631 1820 tablet_bootstrap.cc:492] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap complete.
I20240502 08:18:20.542985 1820 ts_tablet_manager.cc:1367] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.254s user 0.047s sys 0.005s
I20240502 08:18:20.544082 1820 raft_consensus.cc:348] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:20.544457 1820 raft_consensus.cc:729] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: LEARNER
I20240502 08:18:20.544711 1820 consensus_queue.cc:260] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:20.545979 1820 ts_tablet_manager.cc:1398] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240502 08:18:20.546660 1912 tablet_replica.cc:330] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:20.548660 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 1f0bf4bc3c5146539146b825b048bff3-09aa1b5d99114b3095be08c9547fbb54 received from {username='slave'} at 127.20.207.5:54501
I20240502 08:18:20.548768 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 1f0bf4bc3c5146539146b825b048bff3-09aa1b5d99114b3095be08c9547fbb54 on tablet 09aa1b5d99114b3095be08c9547fbb54 with peer 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:18:20.549167 1912 raft_consensus.cc:2229] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:20.549414 1912 raft_consensus.cc:2258] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:20.550853 1912 ts_tablet_manager.cc:1875] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:20.559192 30290 raft_consensus.cc:1261] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:20.559648 1311 consensus_queue.cc:1035] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:20.561380 1912 ts_tablet_manager.cc:1888] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:20.561488 1912 log.cc:1196] T 2066c7aa052342a1aca3eb849aa251fc P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/2066c7aa052342a1aca3eb849aa251fc
I20240502 08:18:20.562542 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet 2066c7aa052342a1aca3eb849aa251fc (table rebalancer_extra_table_20 [id=aff097c8fa414c8aa9c675914ae4280a]) successfully deleted
I20240502 08:18:20.562872 30028 raft_consensus.cc:1261] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:20.563400 1311 consensus_queue.cc:1035] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:20.571993 27255 consensus_queue.cc:237] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:20.576570 30289 raft_consensus.cc:1203] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:20.576787 27114 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet 50cdc758e790419e91c4ebacf9307c85 (table rebalancer_dynamic_table [id=bdd7dc8b8bdf4373b3c4d2c3e428e2ae]). Expected version 1 got 0
I20240502 08:18:20.578516 27388 raft_consensus.cc:1261] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:20.584369 30418 raft_consensus.cc:1261] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:20.588249 1787 consensus_queue.cc:1035] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:20.588835 27114 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet 50cdc758e790419e91c4ebacf9307c85 (table rebalancer_dynamic_table [id=bdd7dc8b8bdf4373b3c4d2c3e428e2ae]). Expected version 1 got 0
I20240502 08:18:20.588922 1787 consensus_queue.cc:1035] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:20.589159 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: feb9cd278950480fb63be0be950d190b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:20.589839 1919 tablet.cc:1722] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:20.590471 1919 tablet.cc:1620] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:20.592727 1856 ts_tablet_manager.cc:902] T 70f4c40bc22947468dfd15804a4e66cd P 83e171a5e16c4008923f1d43fc5ce7b1: Initiating tablet copy from peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:20.594066 27114 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet 50cdc758e790419e91c4ebacf9307c85 (table rebalancer_dynamic_table [id=bdd7dc8b8bdf4373b3c4d2c3e428e2ae]). Expected version 1 got 0
I20240502 08:18:20.595554 1918 tablet.cc:1722] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:20.596040 1918 tablet.cc:1620] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442: MemRowSet was empty: no flush needed.
I20240502 08:18:20.599031 1920 tablet.cc:1722] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:20.599610 1920 tablet.cc:1620] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:20.599612 1551 raft_consensus.cc:2932] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:20.603550 27110 catalog_manager.cc:5576] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815) does not have the latest schema for tablet 50cdc758e790419e91c4ebacf9307c85 (table rebalancer_dynamic_table [id=bdd7dc8b8bdf4373b3c4d2c3e428e2ae]). Expected version 1 got 0
I20240502 08:18:20.603996 27388 raft_consensus.cc:2932] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:20.604627 27113 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=bdd7dc8b8bdf4373b3c4d2c3e428e2ae] alter complete (version 1)
I20240502 08:18:20.610546 30418 raft_consensus.cc:2932] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:20.613011 27114 catalog_manager.cc:5484] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:20.615985 1704 raft_consensus.cc:1050] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1: attempting to promote NON_VOTER 84cda20738d2452da166e31fc6da57a6 to VOTER
I20240502 08:18:20.616724 1704 consensus_queue.cc:237] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:20.619604 30157 raft_consensus.cc:1261] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:20.620407 1695 consensus_queue.cc:1035] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:20.620994 27255 raft_consensus.cc:1261] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:20.621892 1695 consensus_queue.cc:1035] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:20.627902 30418 raft_consensus.cc:1261] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:20.628902 1695 consensus_queue.cc:1035] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:20.635890 30025 raft_consensus.cc:1203] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:18:20.636358 1866 consensus_queue.cc:1035] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:20.644429 30290 raft_consensus.cc:1203] T eca2618572fe4900bd420bb59c22381f P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:20.680117 1704 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:20.684834 30423 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:20.687091 1856 tablet_copy_client.cc:308] T 70f4c40bc22947468dfd15804a4e66cd P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.5:37815
I20240502 08:18:20.686937 27255 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:20.687878 30048 tablet_copy_service.cc:140] P 1f0bf4bc3c5146539146b825b048bff3: Received BeginTabletCopySession request for tablet 70f4c40bc22947468dfd15804a4e66cd from peer 83e171a5e16c4008923f1d43fc5ce7b1 ({username='slave'} at 127.20.207.2:49697)
I20240502 08:18:20.688078 30048 tablet_copy_service.cc:161] P 1f0bf4bc3c5146539146b825b048bff3: Beginning new tablet copy session on tablet 70f4c40bc22947468dfd15804a4e66cd from peer 83e171a5e16c4008923f1d43fc5ce7b1 at {username='slave'} at 127.20.207.2:49697: session id = 83e171a5e16c4008923f1d43fc5ce7b1-70f4c40bc22947468dfd15804a4e66cd
I20240502 08:18:20.691169 30048 tablet_copy_source_session.cc:205] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:20.694264 1856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f4c40bc22947468dfd15804a4e66cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.698024 30156 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:20.700572 1856 tablet_copy_client.cc:783] T 70f4c40bc22947468dfd15804a4e66cd P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:20.701057 1856 tablet_copy_client.cc:647] T 70f4c40bc22947468dfd15804a4e66cd P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:20.702605 27115 catalog_manager.cc:5484] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:20.710944 27109 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:20.712894 1185 consensus_queue.cc:1035] T 56cb28bc9e264d4882d482499b950349 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:20.716068 1856 tablet_copy_client.cc:523] T 70f4c40bc22947468dfd15804a4e66cd P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:20.719403 1856 tablet_bootstrap.cc:492] T 70f4c40bc22947468dfd15804a4e66cd P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:20.723623 1918 tablet.cc:1722] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:20.724237 1918 tablet.cc:1620] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442: MemRowSet was empty: no flush needed.
I20240502 08:18:20.728072 1919 tablet.cc:1722] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:20.728663 1919 tablet.cc:1620] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:20.732136 1856 tablet_bootstrap.cc:492] T 70f4c40bc22947468dfd15804a4e66cd P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:20.732555 1856 tablet_bootstrap.cc:492] T 70f4c40bc22947468dfd15804a4e66cd P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap complete.
I20240502 08:18:20.732815 1856 ts_tablet_manager.cc:1367] T 70f4c40bc22947468dfd15804a4e66cd P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.013s user 0.013s sys 0.000s
I20240502 08:18:20.733546 1856 raft_consensus.cc:348] T 70f4c40bc22947468dfd15804a4e66cd P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:20.734402 1856 raft_consensus.cc:729] T 70f4c40bc22947468dfd15804a4e66cd P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: LEARNER
I20240502 08:18:20.734638 1856 consensus_queue.cc:260] T 70f4c40bc22947468dfd15804a4e66cd P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:20.735471 1856 ts_tablet_manager.cc:1398] T 70f4c40bc22947468dfd15804a4e66cd P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:20.737210 30048 tablet_copy_service.cc:342] P 1f0bf4bc3c5146539146b825b048bff3: Request end of tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-70f4c40bc22947468dfd15804a4e66cd received from {username='slave'} at 127.20.207.2:49697
I20240502 08:18:20.737358 30048 tablet_copy_service.cc:433] P 1f0bf4bc3c5146539146b825b048bff3: ending tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-70f4c40bc22947468dfd15804a4e66cd on tablet 70f4c40bc22947468dfd15804a4e66cd with peer 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:20.739699 1920 tablet.cc:1722] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:20.740188 1920 tablet.cc:1620] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3: MemRowSet was empty: no flush needed.
I20240502 08:18:20.740227 27388 consensus_queue.cc:237] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:20.742450 30157 raft_consensus.cc:1261] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:20.742971 1704 consensus_queue.cc:1035] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:20.744760 30418 raft_consensus.cc:1261] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:20.745246 1927 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:20.746189 30157 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:20.746683 1695 consensus_queue.cc:1035] T 04a0a58eb52f41bf818e5102996d09e0 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:20.748015 30418 raft_consensus.cc:2932] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:20.756006 27114 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=bdd7dc8b8bdf4373b3c4d2c3e428e2ae] alter complete (version 2)
I20240502 08:18:20.757229 27110 catalog_manager.cc:5484] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 8 to 9, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:20.758667 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 04a0a58eb52f41bf818e5102996d09e0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:20.762445 27237 tablet_service.cc:1514] Processing DeleteTablet for tablet 04a0a58eb52f41bf818e5102996d09e0 with delete_type TABLET_DATA_TOMBSTONED (TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39166
I20240502 08:18:20.778857 1944 tablet_replica.cc:330] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:18:20.779186 1944 raft_consensus.cc:2229] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:20.779422 1944 raft_consensus.cc:2258] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:20.782696 1944 ts_tablet_manager.cc:1875] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:20.789356 1944 ts_tablet_manager.cc:1888] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:18:20.789552 1944 log.cc:1196] T 04a0a58eb52f41bf818e5102996d09e0 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/04a0a58eb52f41bf818e5102996d09e0
I20240502 08:18:20.790354 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet 04a0a58eb52f41bf818e5102996d09e0 (table rebalance_test_table_3 [id=cc946163c76846418f7f80018ed16ce5]) successfully deleted
I20240502 08:18:20.840651 1888 consensus_queue.cc:1035] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:20.848485 1888 consensus_queue.cc:1035] T 556fc9ebf7bd4b859f584d46411c2403 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:20.860852 27116 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:20.867183 1747 consensus_queue.cc:1035] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:20.870939 1918 tablet.cc:1722] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:20.872265 1919 tablet.cc:1722] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:20.875980 1920 tablet.cc:1722] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:20.882100 1960 ts_tablet_manager.cc:902] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6: Initiating tablet copy from peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:20.884363 27114 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table_new_name [id=bdd7dc8b8bdf4373b3c4d2c3e428e2ae] alter complete (version 3)
I20240502 08:18:20.893841 1960 tablet_copy_client.cc:308] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.7:40343
I20240502 08:18:20.894804 30310 tablet_copy_service.cc:140] P 49c5c8f70e494edd9fcda40cc0af6cc4: Received BeginTabletCopySession request for tablet 65ac899c820a43e4ab57454e224edc83 from peer 84cda20738d2452da166e31fc6da57a6 ({username='slave'} at 127.20.207.6:52997)
I20240502 08:18:20.895004 30310 tablet_copy_service.cc:161] P 49c5c8f70e494edd9fcda40cc0af6cc4: Beginning new tablet copy session on tablet 65ac899c820a43e4ab57454e224edc83 from peer 84cda20738d2452da166e31fc6da57a6 at {username='slave'} at 127.20.207.6:52997: session id = 84cda20738d2452da166e31fc6da57a6-65ac899c820a43e4ab57454e224edc83
I20240502 08:18:20.896728 1755 consensus_queue.cc:1035] T 0f26bd4120b64d98bd904a4be5297823 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:20.897756 30310 tablet_copy_source_session.cc:205] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:20.901968 1960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65ac899c820a43e4ab57454e224edc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.910717 1960 tablet_copy_client.cc:783] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:20.911057 1960 tablet_copy_client.cc:647] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:20.912811 1960 tablet_copy_client.cc:523] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:20.918282 1960 tablet_bootstrap.cc:492] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:20.922267 30423 raft_consensus.cc:1203] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:18:20.931911 1960 tablet_bootstrap.cc:492] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:20.932339 1960 tablet_bootstrap.cc:492] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6: Bootstrap complete.
I20240502 08:18:20.932579 1960 ts_tablet_manager.cc:1367] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20240502 08:18:20.933339 1960 raft_consensus.cc:348] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:20.933573 1960 raft_consensus.cc:729] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: LEARNER
I20240502 08:18:20.933795 1960 consensus_queue.cc:260] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:20.934427 1960 ts_tablet_manager.cc:1398] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 08:18:20.937203 30310 tablet_copy_service.cc:342] P 49c5c8f70e494edd9fcda40cc0af6cc4: Request end of tablet copy session 84cda20738d2452da166e31fc6da57a6-65ac899c820a43e4ab57454e224edc83 received from {username='slave'} at 127.20.207.6:52997
I20240502 08:18:20.937355 30310 tablet_copy_service.cc:433] P 49c5c8f70e494edd9fcda40cc0af6cc4: ending tablet copy session 84cda20738d2452da166e31fc6da57a6-65ac899c820a43e4ab57454e224edc83 on tablet 65ac899c820a43e4ab57454e224edc83 with peer 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:20.983078 1831 raft_consensus.cc:1050] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3: attempting to promote NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 to VOTER
I20240502 08:18:20.983937 1831 consensus_queue.cc:237] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:20.986373 27516 raft_consensus.cc:1261] T 387e9ef5899541b0be27a41dbcf4475c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:20.987075 27388 raft_consensus.cc:1261] T 387e9ef5899541b0be27a41dbcf4475c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:20.987529 27110 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:20.987660 1847 consensus_queue.cc:1035] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:20.987746 27110 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:20.987869 30157 raft_consensus.cc:1261] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:20.988595 1847 consensus_queue.cc:1035] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:20.989225 1847 consensus_queue.cc:1035] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:20.991112 27110 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: Sending DeleteTablet for 3 replicas of tablet 50cdc758e790419e91c4ebacf9307c85
I20240502 08:18:20.992759 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet 50cdc758e790419e91c4ebacf9307c85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:20 UTC) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:20.993330 27629 tablet_service.cc:1514] Processing DeleteTablet for tablet 50cdc758e790419e91c4ebacf9307c85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:20 UTC) from {username='slave'} at 127.0.0.1:44788
I20240502 08:18:20.993654 30006 tablet_service.cc:1514] Processing DeleteTablet for tablet 50cdc758e790419e91c4ebacf9307c85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:20 UTC) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:21.000466 27116 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:21.001876 1997 tablet_replica.cc:330] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:21.002908 1997 raft_consensus.cc:2229] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:21.003247 1997 raft_consensus.cc:2258] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:21.004206 1997 ts_tablet_manager.cc:1875] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:21.009388 1996 tablet_replica.cc:330] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:18:21.012319 1997 ts_tablet_manager.cc:1888] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:21.012516 1997 log.cc:1196] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/50cdc758e790419e91c4ebacf9307c85
I20240502 08:18:21.009610 1996 raft_consensus.cc:2229] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:21.013187 1996 raft_consensus.cc:2258] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:21.015560 1997 ts_tablet_manager.cc:1909] T 50cdc758e790419e91c4ebacf9307c85 P 1f0bf4bc3c5146539146b825b048bff3: Deleting consensus metadata
I20240502 08:18:21.017052 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet 50cdc758e790419e91c4ebacf9307c85 (table rebalancer_dynamic_table_new_name [id=bdd7dc8b8bdf4373b3c4d2c3e428e2ae]) successfully deleted
I20240502 08:18:21.017194 30400 tablet_service.cc:1467] Processing CreateTablet for tablet c30720c1b245438db3eb4bea2b8bd930 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=04082d4941a343ab8e88027c2100774d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.017650 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c30720c1b245438db3eb4bea2b8bd930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.018960 30139 tablet_service.cc:1467] Processing CreateTablet for tablet c30720c1b245438db3eb4bea2b8bd930 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=04082d4941a343ab8e88027c2100774d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.019524 30139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c30720c1b245438db3eb4bea2b8bd930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.021162 1991 raft_consensus.cc:2932] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:21.022106 27237 tablet_service.cc:1467] Processing CreateTablet for tablet c30720c1b245438db3eb4bea2b8bd930 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=04082d4941a343ab8e88027c2100774d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.022337 27238 tablet_service.cc:1467] Processing CreateTablet for tablet e65281f04bee4c299aa019adb454bbf5 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=04082d4941a343ab8e88027c2100774d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.022553 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c30720c1b245438db3eb4bea2b8bd930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.022856 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65281f04bee4c299aa019adb454bbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.024125 30006 tablet_service.cc:1467] Processing CreateTablet for tablet e65281f04bee4c299aa019adb454bbf5 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=04082d4941a343ab8e88027c2100774d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.024471 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65281f04bee4c299aa019adb454bbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.026733 30401 tablet_service.cc:1467] Processing CreateTablet for tablet 51201cc579394a2b819278a5943485a0 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=04082d4941a343ab8e88027c2100774d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.027170 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51201cc579394a2b819278a5943485a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.030368 30138 tablet_service.cc:1467] Processing CreateTablet for tablet 51201cc579394a2b819278a5943485a0 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=04082d4941a343ab8e88027c2100774d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.030776 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51201cc579394a2b819278a5943485a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.035733 1996 ts_tablet_manager.cc:1875] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:21.036337 27498 tablet_service.cc:1467] Processing CreateTablet for tablet e65281f04bee4c299aa019adb454bbf5 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=04082d4941a343ab8e88027c2100774d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.036718 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65281f04bee4c299aa019adb454bbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.043787 30157 raft_consensus.cc:2932] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:21.044309 2001 tablet_bootstrap.cc:492] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:21.046021 2001 tablet_bootstrap.cc:654] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.048362 27387 raft_consensus.cc:2932] T 387e9ef5899541b0be27a41dbcf4475c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:21.057176 2001 tablet_bootstrap.cc:492] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:21.058652 2001 ts_tablet_manager.cc:1367] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20240502 08:18:21.059605 2001 raft_consensus.cc:348] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.059829 1866 raft_consensus.cc:1050] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 to VOTER
I20240502 08:18:21.060770 2001 raft_consensus.cc:374] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.061029 2001 raft_consensus.cc:729] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.060859 1866 consensus_queue.cc:237] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:21.061067 27116 catalog_manager.cc:5484] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:21.061612 2001 consensus_queue.cc:260] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.066367 2001 ts_tablet_manager.cc:1398] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 08:18:21.077337 27255 raft_consensus.cc:1261] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:21.078341 27652 raft_consensus.cc:1261] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:21.078416 27629 tablet_service.cc:1467] Processing CreateTablet for tablet 51201cc579394a2b819278a5943485a0 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=04082d4941a343ab8e88027c2100774d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.078847 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51201cc579394a2b819278a5943485a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.078948 1729 consensus_queue.cc:1035] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:21.079265 30290 raft_consensus.cc:1261] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:21.079515 1729 consensus_queue.cc:1035] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:21.081015 1729 consensus_queue.cc:1035] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:21.084903 2001 tablet_bootstrap.cc:492] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:21.085037 27387 raft_consensus.cc:1203] T 70f4c40bc22947468dfd15804a4e66cd P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:21.086270 2001 tablet_bootstrap.cc:654] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.090193 27516 raft_consensus.cc:2932] T 387e9ef5899541b0be27a41dbcf4475c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:21.099077 1996 ts_tablet_manager.cc:1888] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:21.100525 30025 consensus_queue.cc:237] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:21.102685 27389 raft_consensus.cc:1261] T 387e9ef5899541b0be27a41dbcf4475c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:21.103308 1847 consensus_queue.cc:1035] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:21.104187 1996 log.cc:1196] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/50cdc758e790419e91c4ebacf9307c85
I20240502 08:18:21.105244 1996 ts_tablet_manager.cc:1909] T 50cdc758e790419e91c4ebacf9307c85 P bf8ac71b15f740a0a928911360773442: Deleting consensus metadata
I20240502 08:18:21.106199 27114 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:21.109722 1995 tablet_replica.cc:330] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:21.110028 1995 raft_consensus.cc:2229] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:21.110316 1995 raft_consensus.cc:2258] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:21.115396 1995 ts_tablet_manager.cc:1875] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:21.116106 2003 tablet_bootstrap.cc:492] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:21.117667 2004 tablet_bootstrap.cc:492] T e65281f04bee4c299aa019adb454bbf5 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:21.117784 2003 tablet_bootstrap.cc:654] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.128360 1847 raft_consensus.cc:2932] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:21.129624 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 56d2a9f088564a66bf6662a4ddbb8145 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1cec00d0ae314fc5b9fbfaa328df9beb]), partition=
I20240502 08:18:21.129981 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56d2a9f088564a66bf6662a4ddbb8145. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.132761 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 387e9ef5899541b0be27a41dbcf4475c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:21.134717 2004 tablet_bootstrap.cc:654] T e65281f04bee4c299aa019adb454bbf5 P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.136636 27500 tablet_service.cc:1467] Processing CreateTablet for tablet 56d2a9f088564a66bf6662a4ddbb8145 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1cec00d0ae314fc5b9fbfaa328df9beb]), partition=
I20240502 08:18:21.136945 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56d2a9f088564a66bf6662a4ddbb8145. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.136651 27388 raft_consensus.cc:2932] T 387e9ef5899541b0be27a41dbcf4475c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:21.145419 27516 raft_consensus.cc:1261] T 387e9ef5899541b0be27a41dbcf4475c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:21.146140 1847 consensus_queue.cc:1035] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:21.148398 27116 catalog_manager.cc:5484] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:21.154613 2008 tablet_bootstrap.cc:492] T e65281f04bee4c299aa019adb454bbf5 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:21.158320 2012 tablet_bootstrap.cc:492] T c30720c1b245438db3eb4bea2b8bd930 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:21.163684 30139 tablet_service.cc:1514] Processing DeleteTablet for tablet 387e9ef5899541b0be27a41dbcf4475c with delete_type TABLET_DATA_TOMBSTONED (TS 84cda20738d2452da166e31fc6da57a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:21.164930 27516 raft_consensus.cc:2932] T 387e9ef5899541b0be27a41dbcf4475c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:21.169247 2004 tablet_bootstrap.cc:492] T e65281f04bee4c299aa019adb454bbf5 P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:21.169463 2004 ts_tablet_manager.cc:1367] T e65281f04bee4c299aa019adb454bbf5 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.000s
I20240502 08:18:21.170670 2004 raft_consensus.cc:348] T e65281f04bee4c299aa019adb454bbf5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.170967 2004 raft_consensus.cc:374] T e65281f04bee4c299aa019adb454bbf5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.171124 2004 raft_consensus.cc:729] T e65281f04bee4c299aa019adb454bbf5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.171424 2004 consensus_queue.cc:260] T e65281f04bee4c299aa019adb454bbf5 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.173512 2004 ts_tablet_manager.cc:1398] T e65281f04bee4c299aa019adb454bbf5 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:21.177238 1995 ts_tablet_manager.cc:1888] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:21.177772 1787 raft_consensus.cc:482] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:21.135649 27631 tablet_service.cc:1467] Processing CreateTablet for tablet 56d2a9f088564a66bf6662a4ddbb8145 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1cec00d0ae314fc5b9fbfaa328df9beb]), partition=
I20240502 08:18:21.177930 1787 raft_consensus.cc:504] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.178483 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56d2a9f088564a66bf6662a4ddbb8145. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.178654 1787 leader_election.cc:288] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:21.178966 2012 tablet_bootstrap.cc:654] T c30720c1b245438db3eb4bea2b8bd930 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.179535 30156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30720c1b245438db3eb4bea2b8bd930" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
W20240502 08:18:21.180107 27194 leader_election.cc:341] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:21.180974 2008 tablet_bootstrap.cc:654] T e65281f04bee4c299aa019adb454bbf5 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.182215 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30720c1b245438db3eb4bea2b8bd930" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
W20240502 08:18:21.185081 27191 leader_election.cc:341] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:21.185240 27191 leader_election.cc:302] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:21.185710 1787 raft_consensus.cc:2726] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:21.187175 2018 tablet_replica.cc:330] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:21.188858 2018 raft_consensus.cc:2229] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:21.190923 2018 raft_consensus.cc:2258] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:21.191277 2001 tablet_bootstrap.cc:492] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:21.191418 2001 ts_tablet_manager.cc:1367] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.107s user 0.003s sys 0.007s
I20240502 08:18:21.192147 2001 raft_consensus.cc:348] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.192353 2001 raft_consensus.cc:374] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.192433 2001 raft_consensus.cc:729] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.192647 2001 consensus_queue.cc:260] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.193239 2001 ts_tablet_manager.cc:1398] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:21.192795 1802 raft_consensus.cc:1050] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6: attempting to promote NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 to VOTER
I20240502 08:18:21.194275 1802 consensus_queue.cc:237] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:21.194557 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 50cdc758e790419e91c4ebacf9307c85 (table rebalancer_dynamic_table_new_name [id=bdd7dc8b8bdf4373b3c4d2c3e428e2ae]) successfully deleted
I20240502 08:18:21.203563 1995 log.cc:1196] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/50cdc758e790419e91c4ebacf9307c85
I20240502 08:18:21.205731 1995 ts_tablet_manager.cc:1909] T 50cdc758e790419e91c4ebacf9307c85 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting consensus metadata
I20240502 08:18:21.207111 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet 50cdc758e790419e91c4ebacf9307c85 (table rebalancer_dynamic_table_new_name [id=bdd7dc8b8bdf4373b3c4d2c3e428e2ae]) successfully deleted
I20240502 08:18:21.207715 2003 tablet_bootstrap.cc:492] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:21.207935 2003 ts_tablet_manager.cc:1367] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.092s user 0.000s sys 0.011s
I20240502 08:18:21.209585 2003 raft_consensus.cc:348] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.210094 2003 raft_consensus.cc:374] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.210278 2003 raft_consensus.cc:729] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.210546 2003 consensus_queue.cc:260] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.211146 2003 ts_tablet_manager.cc:1398] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:18:21.212258 2003 tablet_bootstrap.cc:492] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:21.214068 2003 tablet_bootstrap.cc:654] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.215620 2018 ts_tablet_manager.cc:1875] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:21.225340 1866 raft_consensus.cc:1050] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 to VOTER
I20240502 08:18:21.226148 1866 consensus_queue.cc:237] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:21.230435 1787 raft_consensus.cc:482] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:21.230681 1787 raft_consensus.cc:504] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.232215 1787 leader_election.cc:288] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:21.232731 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65281f04bee4c299aa019adb454bbf5" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:21.232846 2015 tablet_bootstrap.cc:492] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:21.232981 30025 raft_consensus.cc:2454] T e65281f04bee4c299aa019adb454bbf5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:18:21.233531 27191 leader_election.cc:302] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 1f0bf4bc3c5146539146b825b048bff3; no voters:
I20240502 08:18:21.233933 1787 raft_consensus.cc:2781] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:21.234102 1787 raft_consensus.cc:482] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:21.234261 1787 raft_consensus.cc:3037] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.234174 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65281f04bee4c299aa019adb454bbf5" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
W20240502 08:18:21.234743 27194 leader_election.cc:341] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:21.234969 2010 ts_tablet_manager.cc:902] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:21.237546 1787 raft_consensus.cc:504] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.238090 2008 tablet_bootstrap.cc:492] T e65281f04bee4c299aa019adb454bbf5 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:21.238219 2008 ts_tablet_manager.cc:1367] T e65281f04bee4c299aa019adb454bbf5 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.006s
I20240502 08:18:21.238516 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65281f04bee4c299aa019adb454bbf5" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:18:21.241683 30028 raft_consensus.cc:1261] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:21.241909 27257 raft_consensus.cc:1261] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:21.242417 1185 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:21.242892 1185 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:18:21.243153 27194 leader_election.cc:341] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:21.243836 2008 raft_consensus.cc:348] T e65281f04bee4c299aa019adb454bbf5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.243993 2008 raft_consensus.cc:374] T e65281f04bee4c299aa019adb454bbf5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.244045 2008 raft_consensus.cc:729] T e65281f04bee4c299aa019adb454bbf5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.244257 2008 consensus_queue.cc:260] T e65281f04bee4c299aa019adb454bbf5 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.244858 2008 ts_tablet_manager.cc:1398] T e65281f04bee4c299aa019adb454bbf5 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:18:21.245200 2008 tablet_bootstrap.cc:492] T 56d2a9f088564a66bf6662a4ddbb8145 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:21.246651 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65281f04bee4c299aa019adb454bbf5" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:21.246872 30025 raft_consensus.cc:3037] T e65281f04bee4c299aa019adb454bbf5 P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.247727 2008 tablet_bootstrap.cc:654] T 56d2a9f088564a66bf6662a4ddbb8145 P b01e7c8a302d41078fa39cfef665ff59: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.251464 30025 raft_consensus.cc:2454] T e65281f04bee4c299aa019adb454bbf5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:18:21.252106 27191 leader_election.cc:302] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 1f0bf4bc3c5146539146b825b048bff3; no voters: b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:21.252439 1551 raft_consensus.cc:2781] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:21.252585 1551 raft_consensus.cc:686] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:18:21.252890 1551 consensus_queue.cc:237] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.256918 2008 tablet_bootstrap.cc:492] T 56d2a9f088564a66bf6662a4ddbb8145 P b01e7c8a302d41078fa39cfef665ff59: No bootstrap required, opened a new log
I20240502 08:18:21.257400 2008 ts_tablet_manager.cc:1367] T 56d2a9f088564a66bf6662a4ddbb8145 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 08:18:21.258257 2008 raft_consensus.cc:348] T 56d2a9f088564a66bf6662a4ddbb8145 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:21.259997 1787 leader_election.cc:288] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:21.260286 2008 raft_consensus.cc:374] T 56d2a9f088564a66bf6662a4ddbb8145 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.260506 2008 raft_consensus.cc:729] T 56d2a9f088564a66bf6662a4ddbb8145 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.260901 2008 consensus_queue.cc:260] T 56d2a9f088564a66bf6662a4ddbb8145 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:21.262660 30288 raft_consensus.cc:1261] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:21.264278 1185 raft_consensus.cc:2932] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:21.265461 1729 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:21.265583 2008 ts_tablet_manager.cc:1398] T 56d2a9f088564a66bf6662a4ddbb8145 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240502 08:18:21.266767 27258 raft_consensus.cc:2932] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:21.275326 30156 raft_consensus.cc:1203] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:21.277163 27650 raft_consensus.cc:2932] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:21.287787 27116 catalog_manager.cc:5484] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:21.288954 2021 tablet_bootstrap.cc:492] T 56d2a9f088564a66bf6662a4ddbb8145 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:21.290199 27113 catalog_manager.cc:5484] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:21.295444 30289 raft_consensus.cc:1261] T eca2618572fe4900bd420bb59c22381f P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:21.296471 30025 raft_consensus.cc:1261] T eca2618572fe4900bd420bb59c22381f P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:21.297191 30423 raft_consensus.cc:1261] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:21.298238 2012 tablet_bootstrap.cc:492] T c30720c1b245438db3eb4bea2b8bd930 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:21.298380 2012 ts_tablet_manager.cc:1367] T c30720c1b245438db3eb4bea2b8bd930 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.140s user 0.005s sys 0.005s
I20240502 08:18:21.299190 2012 raft_consensus.cc:348] T c30720c1b245438db3eb4bea2b8bd930 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.299398 2012 raft_consensus.cc:374] T c30720c1b245438db3eb4bea2b8bd930 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.299499 2012 raft_consensus.cc:729] T c30720c1b245438db3eb4bea2b8bd930 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.299737 2012 consensus_queue.cc:260] T c30720c1b245438db3eb4bea2b8bd930 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.300334 2012 ts_tablet_manager.cc:1398] T c30720c1b245438db3eb4bea2b8bd930 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:21.300601 2012 tablet_bootstrap.cc:492] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:21.301872 1814 consensus_queue.cc:1035] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:21.308442 2018 ts_tablet_manager.cc:1888] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:21.308646 2018 log.cc:1196] T 387e9ef5899541b0be27a41dbcf4475c P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/387e9ef5899541b0be27a41dbcf4475c
I20240502 08:18:21.310305 2012 tablet_bootstrap.cc:654] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.311682 1814 consensus_queue.cc:1035] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:21.312847 1814 consensus_queue.cc:1035] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:21.319092 2003 tablet_bootstrap.cc:492] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:21.319322 2003 ts_tablet_manager.cc:1367] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.107s user 0.003s sys 0.000s
I20240502 08:18:21.320104 2003 raft_consensus.cc:348] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:21.320408 2003 raft_consensus.cc:374] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.320530 2003 raft_consensus.cc:729] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.320786 2003 consensus_queue.cc:260] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:21.321599 2003 ts_tablet_manager.cc:1398] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:21.323067 2012 tablet_bootstrap.cc:492] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:21.323699 2012 ts_tablet_manager.cc:1367] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.004s
I20240502 08:18:21.323920 2015 tablet_bootstrap.cc:654] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.324762 1814 raft_consensus.cc:482] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:21.324935 1814 raft_consensus.cc:504] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.326522 1814 leader_election.cc:288] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:21.326803 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30720c1b245438db3eb4bea2b8bd930" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:21.327020 27255 raft_consensus.cc:2454] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:21.327510 30095 leader_election.cc:302] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:21.327896 1814 raft_consensus.cc:2781] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:21.328084 1814 raft_consensus.cc:482] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:21.328301 1814 raft_consensus.cc:3037] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.327689 2012 raft_consensus.cc:348] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:21.328636 2012 raft_consensus.cc:374] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.329049 2012 raft_consensus.cc:729] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.330348 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30720c1b245438db3eb4bea2b8bd930" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:21.330566 30423 raft_consensus.cc:2454] T c30720c1b245438db3eb4bea2b8bd930 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:21.330042 2012 consensus_queue.cc:260] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:21.332276 2012 ts_tablet_manager.cc:1398] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240502 08:18:21.332513 1814 raft_consensus.cc:504] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.333834 1814 leader_election.cc:288] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:21.333851 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30720c1b245438db3eb4bea2b8bd930" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:21.334021 30423 raft_consensus.cc:3037] T c30720c1b245438db3eb4bea2b8bd930 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.334275 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30720c1b245438db3eb4bea2b8bd930" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:21.334452 27255 raft_consensus.cc:3037] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.337196 27255 raft_consensus.cc:2454] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:21.337654 30095 leader_election.cc:302] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:21.338933 30423 raft_consensus.cc:2454] T c30720c1b245438db3eb4bea2b8bd930 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:21.341574 2032 raft_consensus.cc:2781] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:21.341814 2032 raft_consensus.cc:686] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:21.342175 2032 consensus_queue.cc:237] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.309619 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet 387e9ef5899541b0be27a41dbcf4475c (table rebalancer_extra_table_8 [id=05e7c365fc854833a30f18097c117d10]) successfully deleted
I20240502 08:18:21.352952 2021 tablet_bootstrap.cc:654] T 56d2a9f088564a66bf6662a4ddbb8145 P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.353662 30290 raft_consensus.cc:2932] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:21.369299 1866 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:21.371248 2015 tablet_bootstrap.cc:492] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:21.371436 2015 ts_tablet_manager.cc:1367] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.139s user 0.004s sys 0.000s
I20240502 08:18:21.371112 30026 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:21.372140 2015 raft_consensus.cc:348] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:21.372656 2015 raft_consensus.cc:374] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.372732 2015 raft_consensus.cc:729] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.372970 2015 consensus_queue.cc:260] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:21.373441 2015 ts_tablet_manager.cc:1398] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:21.373674 2015 tablet_bootstrap.cc:492] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:21.378134 2015 tablet_bootstrap.cc:654] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.385222 30290 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:21.391407 27114 catalog_manager.cc:5484] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:21.392325 2021 tablet_bootstrap.cc:492] T 56d2a9f088564a66bf6662a4ddbb8145 P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:21.392498 2015 tablet_bootstrap.cc:492] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:21.392531 2021 ts_tablet_manager.cc:1367] T 56d2a9f088564a66bf6662a4ddbb8145 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.104s user 0.004s sys 0.000s
I20240502 08:18:21.389101 27116 catalog_manager.cc:5484] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 7 to 8, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:21.393452 2015 ts_tablet_manager.cc:1367] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.005s
I20240502 08:18:21.393354 2021 raft_consensus.cc:348] T 56d2a9f088564a66bf6662a4ddbb8145 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:21.394213 2015 raft_consensus.cc:348] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:21.394469 2015 raft_consensus.cc:374] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.394567 2015 raft_consensus.cc:729] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.394796 2015 consensus_queue.cc:260] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:21.399296 2015 ts_tablet_manager.cc:1398] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 08:18:21.402081 2021 raft_consensus.cc:374] T 56d2a9f088564a66bf6662a4ddbb8145 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.405732 2021 raft_consensus.cc:729] T 56d2a9f088564a66bf6662a4ddbb8145 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.406103 2021 consensus_queue.cc:260] T 56d2a9f088564a66bf6662a4ddbb8145 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:21.406904 2021 ts_tablet_manager.cc:1398] T 56d2a9f088564a66bf6662a4ddbb8145 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 08:18:21.410959 27257 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:21.415016 2033 raft_consensus.cc:2932] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:21.424166 2010 tablet_copy_client.cc:308] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:18:21.426601 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet feb9cd278950480fb63be0be950d190b from peer b01e7c8a302d41078fa39cfef665ff59 ({username='slave'} at 127.20.207.3:51699)
I20240502 08:18:21.426754 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet feb9cd278950480fb63be0be950d190b from peer b01e7c8a302d41078fa39cfef665ff59 at {username='slave'} at 127.20.207.3:51699: session id = b01e7c8a302d41078fa39cfef665ff59-feb9cd278950480fb63be0be950d190b
I20240502 08:18:21.427336 30418 raft_consensus.cc:2932] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:21.430375 27278 tablet_copy_source_session.cc:205] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:21.431490 30289 raft_consensus.cc:2932] T eca2618572fe4900bd420bb59c22381f P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:21.439263 30025 raft_consensus.cc:2932] T eca2618572fe4900bd420bb59c22381f P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:21.442488 1814 raft_consensus.cc:482] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:21.442638 1814 raft_consensus.cc:504] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:21.443353 1814 leader_election.cc:288] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:21.444064 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51201cc579394a2b819278a5943485a0" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:21.444331 27650 raft_consensus.cc:2454] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:21.444988 30096 leader_election.cc:302] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:21.445691 1814 raft_consensus.cc:2781] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:21.445794 1814 raft_consensus.cc:482] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:21.445940 1814 raft_consensus.cc:3037] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.448417 1881 raft_consensus.cc:482] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:21.448588 1881 raft_consensus.cc:504] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:21.449339 1881 leader_election.cc:288] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:21.449823 27518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56d2a9f088564a66bf6662a4ddbb8145" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59" is_pre_election: true
I20240502 08:18:21.450074 27518 raft_consensus.cc:2454] T 56d2a9f088564a66bf6662a4ddbb8145 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:18:21.452415 27587 leader_election.cc:302] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:21.452936 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56d2a9f088564a66bf6662a4ddbb8145" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:21.453212 30290 raft_consensus.cc:2454] T 56d2a9f088564a66bf6662a4ddbb8145 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 0.
I20240502 08:18:21.454077 1881 raft_consensus.cc:2781] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:21.454159 1881 raft_consensus.cc:482] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:21.454210 1881 raft_consensus.cc:3037] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.457094 1881 raft_consensus.cc:504] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:21.460081 1814 raft_consensus.cc:504] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:21.463452 27518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56d2a9f088564a66bf6662a4ddbb8145" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:18:21.463625 27518 raft_consensus.cc:3037] T 56d2a9f088564a66bf6662a4ddbb8145 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.467077 1814 leader_election.cc:288] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:21.468158 2010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feb9cd278950480fb63be0be950d190b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.471809 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51201cc579394a2b819278a5943485a0" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:21.472043 30422 raft_consensus.cc:3037] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.472564 30419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51201cc579394a2b819278a5943485a0" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:21.474684 30422 raft_consensus.cc:2454] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:21.475167 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51201cc579394a2b819278a5943485a0" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:21.475471 27650 raft_consensus.cc:3037] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.410109 27516 consensus_queue.cc:237] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:21.480589 27257 raft_consensus.cc:1261] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:21.481321 1866 consensus_queue.cc:1035] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:21.481712 27116 catalog_manager.cc:5484] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:21.492820 27650 raft_consensus.cc:2454] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:21.493319 27518 raft_consensus.cc:2454] T 56d2a9f088564a66bf6662a4ddbb8145 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:21.493296 30096 leader_election.cc:302] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:21.493788 2032 raft_consensus.cc:2781] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:21.493952 2032 raft_consensus.cc:686] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:21.494203 2032 consensus_queue.cc:237] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } }
I20240502 08:18:21.495589 27587 leader_election.cc:302] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:21.495836 1311 raft_consensus.cc:2781] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:21.496580 30289 raft_consensus.cc:1261] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:21.497614 1866 consensus_queue.cc:1035] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:21.498214 1925 raft_consensus.cc:1050] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad to VOTER
I20240502 08:18:21.499138 1925 consensus_queue.cc:237] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:21.500142 1866 raft_consensus.cc:2932] T fef4f56761bb458c8d84807feb38a617 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:21.500469 1311 raft_consensus.cc:686] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:18:21.500732 1311 consensus_queue.cc:237] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:21.501474 1881 leader_election.cc:288] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 1 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:21.503041 27650 raft_consensus.cc:1261] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:21.503043 30419 raft_consensus.cc:1261] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:21.503587 1551 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:21.504302 1551 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:21.504776 30290 raft_consensus.cc:1261] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:21.508466 1551 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:21.509286 2010 tablet_copy_client.cc:783] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:21.509799 2010 tablet_copy_client.cc:647] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:21.511704 2010 tablet_copy_client.cc:523] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:21.513942 30288 raft_consensus.cc:2932] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:21.513988 27111 catalog_manager.cc:5484] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:21.517639 2010 tablet_bootstrap.cc:492] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:21.522754 30157 consensus_queue.cc:237] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:21.524075 1551 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:21.527887 30028 raft_consensus.cc:1261] T eca2618572fe4900bd420bb59c22381f P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:21.528174 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fef4f56761bb458c8d84807feb38a617 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:21.528810 2032 consensus_queue.cc:1035] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:21.529279 30290 raft_consensus.cc:1261] T eca2618572fe4900bd420bb59c22381f P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:21.529142 30419 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:21.529917 2032 consensus_queue.cc:1035] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:21.530200 27112 catalog_manager.cc:5484] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 7 to 8, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:21.531186 1814 raft_consensus.cc:2932] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:21.531751 30028 raft_consensus.cc:2932] T eca2618572fe4900bd420bb59c22381f P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:21.532418 30288 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:21.535113 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eca2618572fe4900bd420bb59c22381f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:21.538856 30290 raft_consensus.cc:2932] T eca2618572fe4900bd420bb59c22381f P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:21.538965 27116 catalog_manager.cc:5484] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:21.542711 30400 tablet_service.cc:1514] Processing DeleteTablet for tablet eca2618572fe4900bd420bb59c22381f with delete_type TABLET_DATA_TOMBSTONED (TS 31a1e33dcdbf42f58ea72ce1c0afb0ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:21.544749 30289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56d2a9f088564a66bf6662a4ddbb8145" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:21.544946 30289 raft_consensus.cc:3037] T 56d2a9f088564a66bf6662a4ddbb8145 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.550268 2061 tablet_replica.cc:330] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:21.550544 2061 raft_consensus.cc:2229] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:21.551136 2061 raft_consensus.cc:2258] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:21.552800 2061 ts_tablet_manager.cc:1875] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:21.552502 27257 raft_consensus.cc:2932] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:21.558894 27109 catalog_manager.cc:5484] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:21.559697 2061 ts_tablet_manager.cc:1888] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:21.559883 2061 log.cc:1196] T eca2618572fe4900bd420bb59c22381f P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/eca2618572fe4900bd420bb59c22381f
I20240502 08:18:21.561969 30289 raft_consensus.cc:2454] T 56d2a9f088564a66bf6662a4ddbb8145 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:21.563198 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet eca2618572fe4900bd420bb59c22381f (table rebalancer_extra_table_18 [id=2c7cf528be404fbca449af1fcf7d0e23]) successfully deleted
I20240502 08:18:21.572396 27113 catalog_manager.cc:5484] T fef4f56761bb458c8d84807feb38a617 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 5 to 6, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:21.573570 2010 tablet_bootstrap.cc:492] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:21.577001 27629 tablet_service.cc:1514] Processing DeleteTablet for tablet fef4f56761bb458c8d84807feb38a617 with delete_type TABLET_DATA_TOMBSTONED (TS bf8ac71b15f740a0a928911360773442 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44788
I20240502 08:18:21.577555 1996 tablet_replica.cc:330] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:18:21.577728 1996 raft_consensus.cc:2229] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:21.577905 1996 raft_consensus.cc:2258] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:21.579694 2010 tablet_bootstrap.cc:492] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59: Bootstrap complete.
I20240502 08:18:21.580036 2010 ts_tablet_manager.cc:1367] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.062s user 0.002s sys 0.006s
I20240502 08:18:21.580166 1996 ts_tablet_manager.cc:1875] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:21.580847 2010 raft_consensus.cc:348] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:21.581087 2010 raft_consensus.cc:729] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: LEARNER
I20240502 08:18:21.581306 2010 consensus_queue.cc:260] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:21.581828 2010 ts_tablet_manager.cc:1398] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:21.582403 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session b01e7c8a302d41078fa39cfef665ff59-feb9cd278950480fb63be0be950d190b received from {username='slave'} at 127.20.207.3:51699
I20240502 08:18:21.582530 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session b01e7c8a302d41078fa39cfef665ff59-feb9cd278950480fb63be0be950d190b on tablet feb9cd278950480fb63be0be950d190b with peer b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:21.586712 27652 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:21.589476 1996 ts_tablet_manager.cc:1888] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:21.599735 27516 raft_consensus.cc:1203] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:21.600085 1996 log.cc:1196] T fef4f56761bb458c8d84807feb38a617 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/fef4f56761bb458c8d84807feb38a617
I20240502 08:18:21.601058 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet fef4f56761bb458c8d84807feb38a617 (table rebalance_test_table_11 [id=f7b46161b1f6484f9449a5d5036e221f]) successfully deleted
I20240502 08:18:21.659878 27257 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a793eaaeba89458baf8af7caf3f24c70"
dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47604
I20240502 08:18:21.660108 27257 raft_consensus.cc:595] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:18:21.667912 27115 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:21.689986 1866 raft_consensus.cc:482] T e65281f04bee4c299aa019adb454bbf5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:21.690290 1866 raft_consensus.cc:504] T e65281f04bee4c299aa019adb454bbf5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:21.691175 1866 leader_election.cc:288] T e65281f04bee4c299aa019adb454bbf5 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:21.693068 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65281f04bee4c299aa019adb454bbf5" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:21.693367 30025 raft_consensus.cc:2379] T e65281f04bee4c299aa019adb454bbf5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b01e7c8a302d41078fa39cfef665ff59 in current term 1: Already voted for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in this term.
I20240502 08:18:21.696884 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65281f04bee4c299aa019adb454bbf5" candidate_uuid: "b01e7c8a302d41078fa39cfef665ff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:21.701215 27455 leader_election.cc:302] T e65281f04bee4c299aa019adb454bbf5 P b01e7c8a302d41078fa39cfef665ff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b01e7c8a302d41078fa39cfef665ff59; no voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:18:21.703575 1729 raft_consensus.cc:3037] T e65281f04bee4c299aa019adb454bbf5 P b01e7c8a302d41078fa39cfef665ff59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.704650 1991 raft_consensus.cc:1050] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3: attempting to promote NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 to VOTER
I20240502 08:18:21.705479 1991 consensus_queue.cc:237] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:21.708870 1729 raft_consensus.cc:2726] T e65281f04bee4c299aa019adb454bbf5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:21.713387 27388 raft_consensus.cc:1261] T 70f4c40bc22947468dfd15804a4e66cd P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:21.714076 1847 consensus_queue.cc:1035] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:21.722402 30419 raft_consensus.cc:1261] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:21.723030 1129 consensus_queue.cc:1035] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:21.723459 30290 raft_consensus.cc:1261] T 56d2a9f088564a66bf6662a4ddbb8145 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:21.723944 1311 consensus_queue.cc:1035] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.726408 1787 consensus_queue.cc:1035] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.727414 27516 raft_consensus.cc:1261] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:21.728585 1847 consensus_queue.cc:1035] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:21.752838 1814 consensus_queue.cc:1035] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.758193 1129 raft_consensus.cc:2932] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:21.764026 27518 raft_consensus.cc:2932] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:21.764518 30423 raft_consensus.cc:2932] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:21.775828 1888 raft_consensus.cc:1050] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4: attempting to promote NON_VOTER 84cda20738d2452da166e31fc6da57a6 to VOTER
I20240502 08:18:21.776115 27109 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet 56d2a9f088564a66bf6662a4ddbb8145 (table rebalancer_dynamic_table [id=1cec00d0ae314fc5b9fbfaa328df9beb]). Expected version 1 got 0
I20240502 08:18:21.776721 1888 consensus_queue.cc:237] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:21.781420 30025 raft_consensus.cc:1261] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:21.782332 1909 consensus_queue.cc:1035] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:21.782507 30157 raft_consensus.cc:1261] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:21.779563 27115 catalog_manager.cc:5484] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:21.785151 1909 consensus_queue.cc:1035] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:21.787459 2107 tablet.cc:1722] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:21.788023 1551 raft_consensus.cc:979] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: : Instructing follower 49c5c8f70e494edd9fcda40cc0af6cc4 to start an election
I20240502 08:18:21.788172 1551 raft_consensus.cc:1067] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Signalling peer 49c5c8f70e494edd9fcda40cc0af6cc4 to start an election
I20240502 08:18:21.789346 30288 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a793eaaeba89458baf8af7caf3f24c70"
dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
from {username='slave'} at 127.20.207.1:49483
I20240502 08:18:21.789593 30288 raft_consensus.cc:482] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:21.789701 30288 raft_consensus.cc:3037] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:21.792498 30288 raft_consensus.cc:504] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:21.794173 30288 leader_election.cc:288] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 2 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:21.796402 27109 catalog_manager.cc:5576] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309) does not have the latest schema for tablet 56d2a9f088564a66bf6662a4ddbb8145 (table rebalancer_dynamic_table [id=1cec00d0ae314fc5b9fbfaa328df9beb]). Expected version 1 got 0
I20240502 08:18:21.796412 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a793eaaeba89458baf8af7caf3f24c70" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:21.796607 27257 raft_consensus.cc:3032] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:21.796713 27257 raft_consensus.cc:729] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:18:21.797042 27257 consensus_queue.cc:260] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:21.797542 27257 raft_consensus.cc:3037] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:21.798184 2107 tablet.cc:1620] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442: MemRowSet was empty: no flush needed.
I20240502 08:18:21.800508 27257 raft_consensus.cc:2454] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 2.
I20240502 08:18:21.801216 1814 consensus_queue.cc:1035] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.802130 30156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a793eaaeba89458baf8af7caf3f24c70" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:21.802335 30156 raft_consensus.cc:3037] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:21.805409 30156 raft_consensus.cc:2454] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 2.
I20240502 08:18:21.806267 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a793eaaeba89458baf8af7caf3f24c70" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
I20240502 08:18:21.806504 27387 raft_consensus.cc:3037] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:21.806934 30226 leader_election.cc:302] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 49c5c8f70e494edd9fcda40cc0af6cc4, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:21.807229 2109 raft_consensus.cc:2781] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:21.807363 2109 raft_consensus.cc:686] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 LEADER]: Becoming Leader. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:18:21.807694 2109 consensus_queue.cc:237] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:21.810261 27387 raft_consensus.cc:2454] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 2.
I20240502 08:18:21.811865 2115 tablet.cc:1722] T 56d2a9f088564a66bf6662a4ddbb8145 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:21.814697 2115 tablet.cc:1620] T 56d2a9f088564a66bf6662a4ddbb8145 P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:21.815692 27109 catalog_manager.cc:5576] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343) does not have the latest schema for tablet 56d2a9f088564a66bf6662a4ddbb8145 (table rebalancer_dynamic_table [id=1cec00d0ae314fc5b9fbfaa328df9beb]). Expected version 1 got 0
I20240502 08:18:21.823213 1894 raft_consensus.cc:2932] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:21.830988 30025 raft_consensus.cc:2932] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:21.838265 27115 catalog_manager.cc:5484] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: term changed from 1 to 2, leader changed from 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) to 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7). New cstate: current_term: 2 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:21.839781 27389 raft_consensus.cc:2932] T 70f4c40bc22947468dfd15804a4e66cd P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:21.840117 30157 raft_consensus.cc:2932] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:21.851521 27113 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=1cec00d0ae314fc5b9fbfaa328df9beb] alter complete (version 1)
I20240502 08:18:21.859244 27112 catalog_manager.cc:5484] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 2 to 3, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:21.873649 1925 consensus_queue.cc:1035] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.897585 2032 consensus_queue.cc:1035] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.900758 27516 raft_consensus.cc:1261] T 56d2a9f088564a66bf6662a4ddbb8145 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:21.900951 2032 consensus_queue.cc:1035] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.924633 27648 raft_consensus.cc:1261] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:21.924773 2122 consensus_queue.cc:1035] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.928732 1888 consensus_queue.cc:1035] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 08:18:21.952955 27326 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.3:44469, user_credentials={real_user=slave}} blocked reactor thread for 158458us
I20240502 08:18:21.965934 2131 tablet.cc:1722] T 56d2a9f088564a66bf6662a4ddbb8145 P b01e7c8a302d41078fa39cfef665ff59: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:21.966392 2131 tablet.cc:1620] T 56d2a9f088564a66bf6662a4ddbb8145 P b01e7c8a302d41078fa39cfef665ff59: MemRowSet was empty: no flush needed.
I20240502 08:18:21.975314 27652 raft_consensus.cc:2932] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:21.986672 27113 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34332:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:21.994982 2107 tablet.cc:1722] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:21.995435 2107 tablet.cc:1620] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442: MemRowSet was empty: no flush needed.
I20240502 08:18:21.997937 2115 tablet.cc:1722] T 56d2a9f088564a66bf6662a4ddbb8145 P 49c5c8f70e494edd9fcda40cc0af6cc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:21.998433 2115 tablet.cc:1620] T 56d2a9f088564a66bf6662a4ddbb8145 P 49c5c8f70e494edd9fcda40cc0af6cc4: MemRowSet was empty: no flush needed.
I20240502 08:18:22.009214 27111 catalog_manager.cc:5576] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469) does not have the latest schema for tablet 56d2a9f088564a66bf6662a4ddbb8145 (table rebalancer_dynamic_table [id=1cec00d0ae314fc5b9fbfaa328df9beb]). Expected version 2 got 1
I20240502 08:18:22.015209 27113 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c086547273ea42f0a31343f9f60d13f0: rebalancer_dynamic_table [id=1cec00d0ae314fc5b9fbfaa328df9beb] alter complete (version 2)
I20240502 08:18:22.016566 30025 consensus_queue.cc:237] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:22.019546 27389 raft_consensus.cc:1261] T 70f4c40bc22947468dfd15804a4e66cd P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:22.020745 1991 consensus_queue.cc:1035] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:22.022449 30419 raft_consensus.cc:1261] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:22.027477 1129 consensus_queue.cc:1035] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:22.028707 1991 raft_consensus.cc:2932] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:22.033239 30419 raft_consensus.cc:2932] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:22.035917 27388 raft_consensus.cc:2932] T 70f4c40bc22947468dfd15804a4e66cd P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:22.041158 27113 catalog_manager.cc:5484] T 70f4c40bc22947468dfd15804a4e66cd P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 3 to 4, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:22.050766 27498 tablet_service.cc:1514] Processing DeleteTablet for tablet 70f4c40bc22947468dfd15804a4e66cd with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:22.050912 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 70f4c40bc22947468dfd15804a4e66cd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:22.053665 27115 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:22.067009 2131 tablet.cc:1722] T 56d2a9f088564a66bf6662a4ddbb8145 P b01e7c8a302d41078fa39cfef665ff59: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:22.067452 2131 tablet.cc:1620] T 56d2a9f088564a66bf6662a4ddbb8145 P b01e7c8a302d41078fa39cfef665ff59: MemRowSet was empty: no flush needed.
I20240502 08:18:22.071516 2135 tablet_replica.cc:330] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:22.071694 2135 raft_consensus.cc:2229] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:22.072077 2135 raft_consensus.cc:2258] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:22.075255 30139 tablet_service.cc:1467] Processing CreateTablet for tablet d46a094599854588a411f168ee3a262c (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=a24d262efce245328528f2b42605f575]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:22.075636 30139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d46a094599854588a411f168ee3a262c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.076074 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 3bfa383343184f18969fa52ed97ab07c (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=a24d262efce245328528f2b42605f575]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:22.076472 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bfa383343184f18969fa52ed97ab07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.076470 2135 ts_tablet_manager.cc:1875] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:22.081506 30006 tablet_service.cc:1467] Processing CreateTablet for tablet d46a094599854588a411f168ee3a262c (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=a24d262efce245328528f2b42605f575]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:22.081897 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d46a094599854588a411f168ee3a262c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.082345 27629 tablet_service.cc:1467] Processing CreateTablet for tablet ca8d1416a54344feae081ed7b19dd1de (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=a24d262efce245328528f2b42605f575]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:22.082700 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8d1416a54344feae081ed7b19dd1de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.083715 2135 ts_tablet_manager.cc:1888] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:22.083812 2135 log.cc:1196] T 70f4c40bc22947468dfd15804a4e66cd P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/70f4c40bc22947468dfd15804a4e66cd
I20240502 08:18:22.090489 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 3bfa383343184f18969fa52ed97ab07c (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=a24d262efce245328528f2b42605f575]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:22.090688 30270 tablet_service.cc:1467] Processing CreateTablet for tablet ca8d1416a54344feae081ed7b19dd1de (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=a24d262efce245328528f2b42605f575]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:22.090920 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bfa383343184f18969fa52ed97ab07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.091199 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8d1416a54344feae081ed7b19dd1de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.092005 30401 tablet_service.cc:1467] Processing CreateTablet for tablet ca8d1416a54344feae081ed7b19dd1de (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=a24d262efce245328528f2b42605f575]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:22.092437 30401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8d1416a54344feae081ed7b19dd1de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.094381 30400 tablet_service.cc:1467] Processing CreateTablet for tablet d46a094599854588a411f168ee3a262c (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=a24d262efce245328528f2b42605f575]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:22.094743 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d46a094599854588a411f168ee3a262c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.095760 30007 tablet_service.cc:1467] Processing CreateTablet for tablet 3bfa383343184f18969fa52ed97ab07c (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=a24d262efce245328528f2b42605f575]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:22.096117 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bfa383343184f18969fa52ed97ab07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.102923 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 70f4c40bc22947468dfd15804a4e66cd (table rebalancer_extra_table_32 [id=3c7f8803df8a4ae885409a0c0a114f1f]) successfully deleted
I20240502 08:18:22.120016 2140 tablet_bootstrap.cc:492] T d46a094599854588a411f168ee3a262c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:22.122529 2136 tablet_bootstrap.cc:492] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:22.125912 2137 tablet_bootstrap.cc:492] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:22.126320 2140 tablet_bootstrap.cc:654] T d46a094599854588a411f168ee3a262c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:22.134040 2136 tablet_bootstrap.cc:654] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:22.137676 2137 tablet_bootstrap.cc:654] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:22.139624 2136 tablet_bootstrap.cc:492] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442: No bootstrap required, opened a new log
I20240502 08:18:22.139806 2136 ts_tablet_manager.cc:1367] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240502 08:18:22.140530 2140 tablet_bootstrap.cc:492] T d46a094599854588a411f168ee3a262c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:22.140725 2140 ts_tablet_manager.cc:1367] T d46a094599854588a411f168ee3a262c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.002s
I20240502 08:18:22.141613 2136 raft_consensus.cc:348] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:22.142340 2140 raft_consensus.cc:348] T d46a094599854588a411f168ee3a262c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:22.142603 2140 raft_consensus.cc:374] T d46a094599854588a411f168ee3a262c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.142741 2136 raft_consensus.cc:374] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.142923 2140 raft_consensus.cc:729] T d46a094599854588a411f168ee3a262c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.143438 2136 raft_consensus.cc:729] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.144253 2140 consensus_queue.cc:260] T d46a094599854588a411f168ee3a262c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:22.144723 2136 consensus_queue.cc:260] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:22.145772 2145 tablet_bootstrap.cc:492] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:22.148011 2145 tablet_bootstrap.cc:654] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:22.152072 2136 ts_tablet_manager.cc:1398] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240502 08:18:22.152725 2140 ts_tablet_manager.cc:1398] T d46a094599854588a411f168ee3a262c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 08:18:22.154750 2140 tablet_bootstrap.cc:492] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:22.156445 2140 tablet_bootstrap.cc:654] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:22.156852 2144 tablet_bootstrap.cc:492] T 3bfa383343184f18969fa52ed97ab07c P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:22.158468 2144 tablet_bootstrap.cc:654] T 3bfa383343184f18969fa52ed97ab07c P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:22.159399 2137 tablet_bootstrap.cc:492] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: No bootstrap required, opened a new log
I20240502 08:18:22.159592 2137 ts_tablet_manager.cc:1367] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20240502 08:18:22.160310 2137 raft_consensus.cc:348] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:22.160584 2137 raft_consensus.cc:374] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.160708 2137 raft_consensus.cc:729] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.161002 2137 consensus_queue.cc:260] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:22.163813 2137 ts_tablet_manager.cc:1398] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:18:22.165647 2145 tablet_bootstrap.cc:492] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6: No bootstrap required, opened a new log
I20240502 08:18:22.166816 2145 ts_tablet_manager.cc:1367] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20240502 08:18:22.167562 2145 raft_consensus.cc:348] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:22.167793 2145 raft_consensus.cc:374] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.167928 2145 raft_consensus.cc:729] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.168208 2145 consensus_queue.cc:260] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:22.168764 2145 ts_tablet_manager.cc:1398] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:22.178867 2139 tablet_bootstrap.cc:492] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:22.181882 2139 tablet_bootstrap.cc:654] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:22.183197 2140 tablet_bootstrap.cc:492] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad: No bootstrap required, opened a new log
I20240502 08:18:22.183964 2144 tablet_bootstrap.cc:492] T 3bfa383343184f18969fa52ed97ab07c P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:22.184211 2139 tablet_bootstrap.cc:492] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:22.184211 2140 ts_tablet_manager.cc:1367] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.008s
I20240502 08:18:22.184391 2139 ts_tablet_manager.cc:1367] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:18:22.185022 2140 raft_consensus.cc:348] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:22.185027 2139 raft_consensus.cc:348] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:22.185209 2139 raft_consensus.cc:374] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.185241 2140 raft_consensus.cc:374] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.185314 2139 raft_consensus.cc:729] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.185323 2140 raft_consensus.cc:729] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.185573 2144 ts_tablet_manager.cc:1367] T 3bfa383343184f18969fa52ed97ab07c P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.000s
I20240502 08:18:22.185495 2139 consensus_queue.cc:260] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:22.185938 2139 ts_tablet_manager.cc:1398] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:18:22.186283 2139 tablet_bootstrap.cc:492] T 3bfa383343184f18969fa52ed97ab07c P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:22.186254 2144 raft_consensus.cc:348] T 3bfa383343184f18969fa52ed97ab07c P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:22.187530 2140 consensus_queue.cc:260] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:22.188078 2140 ts_tablet_manager.cc:1398] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:22.188755 2144 raft_consensus.cc:374] T 3bfa383343184f18969fa52ed97ab07c P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.188907 2144 raft_consensus.cc:729] T 3bfa383343184f18969fa52ed97ab07c P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.189020 2139 tablet_bootstrap.cc:654] T 3bfa383343184f18969fa52ed97ab07c P 49c5c8f70e494edd9fcda40cc0af6cc4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:22.189137 2144 consensus_queue.cc:260] T 3bfa383343184f18969fa52ed97ab07c P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:22.189692 2144 ts_tablet_manager.cc:1398] T 3bfa383343184f18969fa52ed97ab07c P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:18:22.191614 2139 tablet_bootstrap.cc:492] T 3bfa383343184f18969fa52ed97ab07c P 49c5c8f70e494edd9fcda40cc0af6cc4: No bootstrap required, opened a new log
I20240502 08:18:22.191780 2139 ts_tablet_manager.cc:1367] T 3bfa383343184f18969fa52ed97ab07c P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:18:22.192921 2139 raft_consensus.cc:348] T 3bfa383343184f18969fa52ed97ab07c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:22.193176 2139 raft_consensus.cc:374] T 3bfa383343184f18969fa52ed97ab07c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.193289 2139 raft_consensus.cc:729] T 3bfa383343184f18969fa52ed97ab07c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.193567 2139 consensus_queue.cc:260] T 3bfa383343184f18969fa52ed97ab07c P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:22.194124 2139 ts_tablet_manager.cc:1398] T 3bfa383343184f18969fa52ed97ab07c P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:22.194624 2144 tablet_bootstrap.cc:492] T d46a094599854588a411f168ee3a262c P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:22.196125 2144 tablet_bootstrap.cc:654] T d46a094599854588a411f168ee3a262c P 1f0bf4bc3c5146539146b825b048bff3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:22.210085 2144 tablet_bootstrap.cc:492] T d46a094599854588a411f168ee3a262c P 1f0bf4bc3c5146539146b825b048bff3: No bootstrap required, opened a new log
I20240502 08:18:22.210394 2144 ts_tablet_manager.cc:1367] T d46a094599854588a411f168ee3a262c P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20240502 08:18:22.211495 2144 raft_consensus.cc:348] T d46a094599854588a411f168ee3a262c P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:22.211814 2144 raft_consensus.cc:374] T d46a094599854588a411f168ee3a262c P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.212129 2144 raft_consensus.cc:729] T d46a094599854588a411f168ee3a262c P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.212457 2144 consensus_queue.cc:260] T d46a094599854588a411f168ee3a262c P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:22.213251 2144 ts_tablet_manager.cc:1398] T d46a094599854588a411f168ee3a262c P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:18:22.223465 1787 raft_consensus.cc:1050] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER b01e7c8a302d41078fa39cfef665ff59 to VOTER
I20240502 08:18:22.224329 1787 consensus_queue.cc:237] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:22.236052 30419 raft_consensus.cc:1261] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:22.236563 2056 consensus_queue.cc:1035] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:22.236781 27518 raft_consensus.cc:1261] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:22.238332 27388 raft_consensus.cc:1261] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:22.238395 2056 consensus_queue.cc:1035] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:22.238881 2056 consensus_queue.cc:1035] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:22.248639 1551 raft_consensus.cc:482] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:22.248785 1551 raft_consensus.cc:504] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:22.249315 1551 leader_election.cc:288] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:22.250123 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bfa383343184f18969fa52ed97ab07c" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" is_pre_election: true
I20240502 08:18:22.250329 30288 raft_consensus.cc:2454] T 3bfa383343184f18969fa52ed97ab07c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:18:22.250633 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bfa383343184f18969fa52ed97ab07c" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:22.250815 30025 raft_consensus.cc:2454] T 3bfa383343184f18969fa52ed97ab07c P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 0.
I20240502 08:18:22.251444 27191 leader_election.cc:302] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:18:22.251869 1551 raft_consensus.cc:2781] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:22.251948 1551 raft_consensus.cc:482] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:22.252010 1551 raft_consensus.cc:3037] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.252311 2032 raft_consensus.cc:482] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:22.252487 2032 raft_consensus.cc:504] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:22.253244 2032 leader_election.cc:288] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:22.254433 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46a094599854588a411f168ee3a262c" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:22.254679 30423 raft_consensus.cc:2454] T d46a094599854588a411f168ee3a262c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 0.
I20240502 08:18:22.255426 1551 raft_consensus.cc:504] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:22.256181 30093 leader_election.cc:302] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:22.256263 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bfa383343184f18969fa52ed97ab07c" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:22.256425 30288 raft_consensus.cc:3037] T 3bfa383343184f18969fa52ed97ab07c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.256707 2033 raft_consensus.cc:2781] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:22.256827 2033 raft_consensus.cc:482] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:22.256916 2033 raft_consensus.cc:3037] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.259140 30288 raft_consensus.cc:2454] T 3bfa383343184f18969fa52ed97ab07c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:18:22.260012 27191 leader_election.cc:302] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 49c5c8f70e494edd9fcda40cc0af6cc4; no voters:
I20240502 08:18:22.260242 1925 raft_consensus.cc:2781] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:22.260347 1925 raft_consensus.cc:686] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:18:22.260579 1925 consensus_queue.cc:237] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:22.263398 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bfa383343184f18969fa52ed97ab07c" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:22.263609 30028 raft_consensus.cc:3037] T 3bfa383343184f18969fa52ed97ab07c P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.264835 2033 raft_consensus.cc:504] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:22.266634 30026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46a094599854588a411f168ee3a262c" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:22.266770 30026 raft_consensus.cc:3037] T d46a094599854588a411f168ee3a262c P 1f0bf4bc3c5146539146b825b048bff3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.268618 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46a094599854588a411f168ee3a262c" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:22.269788 30028 raft_consensus.cc:2454] T 3bfa383343184f18969fa52ed97ab07c P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:18:22.270452 1551 leader_election.cc:288] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 1 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:22.270440 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46a094599854588a411f168ee3a262c" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:22.270716 30423 raft_consensus.cc:3037] T d46a094599854588a411f168ee3a262c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.272464 30026 raft_consensus.cc:2454] T d46a094599854588a411f168ee3a262c P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:22.274325 30093 leader_election.cc:302] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:22.274983 2032 raft_consensus.cc:2781] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:22.275197 2032 raft_consensus.cc:686] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:22.275465 2032 consensus_queue.cc:237] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:22.277591 30423 raft_consensus.cc:2454] T d46a094599854588a411f168ee3a262c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:22.278239 2033 leader_election.cc:288] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 1 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:22.278195 2056 raft_consensus.cc:2932] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:22.283675 27112 catalog_manager.cc:5484] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:22.284008 27516 raft_consensus.cc:2932] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:22.284781 27389 raft_consensus.cc:1261] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 2 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:22.285331 1888 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:22.296149 27112 catalog_manager.cc:5484] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:22.302318 27389 raft_consensus.cc:2932] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:22.310760 27112 catalog_manager.cc:5484] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:22.315518 30419 raft_consensus.cc:2932] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:22.325102 27255 consensus_queue.cc:237] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:22.326889 27518 raft_consensus.cc:1261] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:22.328634 1551 consensus_queue.cc:1035] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:22.331068 30419 raft_consensus.cc:1261] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:22.331569 1551 consensus_queue.cc:1035] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:22.333633 1551 raft_consensus.cc:2932] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:22.334338 30419 raft_consensus.cc:2932] T feb9cd278950480fb63be0be950d190b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:22.336081 27518 raft_consensus.cc:2932] T feb9cd278950480fb63be0be950d190b P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:22.339612 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet feb9cd278950480fb63be0be950d190b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:22.344295 27112 catalog_manager.cc:5484] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 3 to 4, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:22.348925 27366 tablet_service.cc:1514] Processing DeleteTablet for tablet feb9cd278950480fb63be0be950d190b with delete_type TABLET_DATA_TOMBSTONED (TS 83e171a5e16c4008923f1d43fc5ce7b1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40868
I20240502 08:18:22.349545 2179 tablet_replica.cc:330] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:18:22.349821 2179 raft_consensus.cc:2229] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:22.350175 2179 raft_consensus.cc:2258] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:22.351547 2179 ts_tablet_manager.cc:1875] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:22.357844 2179 ts_tablet_manager.cc:1888] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:22.357990 2179 log.cc:1196] T feb9cd278950480fb63be0be950d190b P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/feb9cd278950480fb63be0be950d190b
I20240502 08:18:22.358834 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet feb9cd278950480fb63be0be950d190b (table rebalancer_extra_table_14 [id=95b099178a3c41f7b5f6e2f30dfe9f42]) successfully deleted
I20240502 08:18:22.364795 27255 consensus_queue.cc:237] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:22.366705 30423 raft_consensus.cc:1261] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:22.367204 1551 consensus_queue.cc:1035] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:22.368939 27652 raft_consensus.cc:1261] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:22.369037 1894 raft_consensus.cc:482] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:22.369186 1894 raft_consensus.cc:504] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:22.369434 1787 consensus_queue.cc:1035] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:22.369836 1894 leader_election.cc:288] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:22.370366 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8d1416a54344feae081ed7b19dd1de" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:22.370433 27649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8d1416a54344feae081ed7b19dd1de" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:22.370558 30423 raft_consensus.cc:2454] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
I20240502 08:18:22.370671 27649 raft_consensus.cc:2454] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 0.
W20240502 08:18:22.370754 27194 consensus_peers.cc:485] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Couldn't send request to peer 84cda20738d2452da166e31fc6da57a6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20a836dd442943d08a87d61afc785cd1. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:22.371003 30226 leader_election.cc:302] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:22.371534 1894 raft_consensus.cc:2781] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:22.371640 1894 raft_consensus.cc:482] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:22.371718 1894 raft_consensus.cc:3037] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.373560 1894 raft_consensus.cc:504] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:22.374121 1894 leader_election.cc:288] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:22.374684 27649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8d1416a54344feae081ed7b19dd1de" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:22.374861 27649 raft_consensus.cc:3037] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.377055 27649 raft_consensus.cc:2454] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:22.377360 30226 leader_election.cc:302] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c5c8f70e494edd9fcda40cc0af6cc4, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:22.377581 1888 raft_consensus.cc:2781] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:22.377707 1888 raft_consensus.cc:686] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:18:22.377961 1888 consensus_queue.cc:237] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } }
I20240502 08:18:22.382661 1551 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:22.384361 30423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8d1416a54344feae081ed7b19dd1de" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:22.384609 30423 raft_consensus.cc:3037] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.384696 30422 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:22.387531 27652 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:22.388221 30423 raft_consensus.cc:2454] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 1.
I20240502 08:18:22.391007 27115 catalog_manager.cc:5484] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7). New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:22.392539 27113 catalog_manager.cc:5484] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:22.417771 27255 raft_consensus.cc:1261] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:22.418404 2190 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:22.430393 30157 raft_consensus.cc:1261] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:22.431921 1894 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:22.455451 30288 consensus_queue.cc:237] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:22.457224 27389 raft_consensus.cc:1261] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 2 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:22.457870 2188 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:22.462535 1894 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:22.470170 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a793eaaeba89458baf8af7caf3f24c70 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:22.470981 27389 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:22.472442 27115 catalog_manager.cc:5484] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 8 to 10, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New cstate: current_term: 2 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:22.480782 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet a793eaaeba89458baf8af7caf3f24c70 with delete_type TABLET_DATA_TOMBSTONED (TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:39166
I20240502 08:18:22.492604 30288 consensus_queue.cc:237] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:22.494509 27652 raft_consensus.cc:1261] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:22.495075 2190 consensus_queue.cc:1035] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:22.497516 30159 raft_consensus.cc:1261] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:22.497388 2190 raft_consensus.cc:2932] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:22.499645 30156 raft_consensus.cc:1261] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:22.500808 2199 tablet_replica.cc:330] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:18:22.503556 1894 consensus_queue.cc:1035] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:22.504699 1894 consensus_queue.cc:1035] T a793eaaeba89458baf8af7caf3f24c70 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:22.506646 30159 raft_consensus.cc:2932] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:22.506745 27652 raft_consensus.cc:2932] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:22.507340 30156 raft_consensus.cc:2932] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:22.513830 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 65ac899c820a43e4ab57454e224edc83 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:22.514915 27112 catalog_manager.cc:5484] T 65ac899c820a43e4ab57454e224edc83 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 3 to 4, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:22.520152 30007 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ac899c820a43e4ab57454e224edc83 with delete_type TABLET_DATA_TOMBSTONED (TS 1f0bf4bc3c5146539146b825b048bff3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:22.528537 2199 raft_consensus.cc:2229] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:22.528851 2199 raft_consensus.cc:2258] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:22.530532 2199 ts_tablet_manager.cc:1875] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:22.535396 2200 tablet_replica.cc:330] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:22.536862 2200 raft_consensus.cc:2229] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:22.537143 2200 raft_consensus.cc:2258] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:22.539191 2200 ts_tablet_manager.cc:1875] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:22.543998 27518 consensus_queue.cc:237] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:22.544920 2199 ts_tablet_manager.cc:1888] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 08:18:22.545071 2199 log.cc:1196] T a793eaaeba89458baf8af7caf3f24c70 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/a793eaaeba89458baf8af7caf3f24c70
I20240502 08:18:22.545804 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet a793eaaeba89458baf8af7caf3f24c70 (table rebalance_test_table_7 [id=83c398ec83c74ebf84113a0334649e63]) successfully deleted
W20240502 08:18:22.546526 27453 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 -> Peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Couldn't send request to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb5345b931504cfdbb272189b719efa4. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:22.546689 27257 raft_consensus.cc:1261] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:22.546940 30156 raft_consensus.cc:1261] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:22.547245 2202 consensus_queue.cc:1035] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:22.547693 1729 consensus_queue.cc:1035] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:22.547767 2200 ts_tablet_manager.cc:1888] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:22.547873 2200 log.cc:1196] T 65ac899c820a43e4ab57454e224edc83 P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/65ac899c820a43e4ab57454e224edc83
I20240502 08:18:22.548595 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet 65ac899c820a43e4ab57454e224edc83 (table rebalancer_extra_table_22 [id=77ecd1036c454af293decc2ea5173f97]) successfully deleted
I20240502 08:18:22.550413 1729 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:22.550942 30156 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:22.552206 27257 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:22.556888 27115 catalog_manager.cc:5484] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 6 to 7, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:22.580133 30422 consensus_queue.cc:237] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:22.582473 30156 raft_consensus.cc:1261] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:22.582609 27518 raft_consensus.cc:1261] T 1adfeb89dc5341a2b8cf962eb28da45e P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:22.583003 2213 consensus_queue.cc:1035] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:22.583472 1747 consensus_queue.cc:1035] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:22.584304 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Couldn't send request to peer bf8ac71b15f740a0a928911360773442. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1adfeb89dc5341a2b8cf962eb28da45e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:22.586632 1747 raft_consensus.cc:2932] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:22.587359 27518 raft_consensus.cc:2932] T 1adfeb89dc5341a2b8cf962eb28da45e P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:22.591632 30157 raft_consensus.cc:2932] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:22.593158 27115 catalog_manager.cc:5484] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:22.614460 30028 consensus_queue.cc:237] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:22.616804 30156 raft_consensus.cc:1261] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:22.617506 1847 consensus_queue.cc:1035] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:22.622432 1991 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:22.623090 30156 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
W20240502 08:18:22.626341 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26545a868bd6415c9c9db4869b2841c7. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:22.626438 27257 raft_consensus.cc:1261] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:22.627050 1847 consensus_queue.cc:1035] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:22.630156 27115 catalog_manager.cc:5484] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 4 to 5, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:22.635885 27257 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:22.671028 27518 consensus_queue.cc:237] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
W20240502 08:18:22.673053 27453 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 -> Peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Couldn't send request to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2776f857a5f47e9b8022639dc54757d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:22.673049 30288 raft_consensus.cc:1261] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:22.673509 27652 raft_consensus.cc:1261] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:22.673477 1729 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:22.674012 1866 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:22.677278 2205 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:22.677839 30288 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:22.680891 27652 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:22.683532 27112 catalog_manager.cc:5484] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 6 to 7, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:22.714769 30156 consensus_queue.cc:237] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:22.717109 27388 raft_consensus.cc:1261] T b9d6207b6bb3436f996a069ff910a56e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:22.717119 30422 raft_consensus.cc:1261] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:22.717703 1814 consensus_queue.cc:1035] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:22.718050 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9d6207b6bb3436f996a069ff910a56e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:22.718291 2033 consensus_queue.cc:1035] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:22.731843 2032 raft_consensus.cc:2932] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:22.732396 27388 raft_consensus.cc:2932] T b9d6207b6bb3436f996a069ff910a56e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:22.740154 27112 catalog_manager.cc:5484] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:22.735963 30422 raft_consensus.cc:2932] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:22.778582 1551 consensus_queue.cc:1035] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:22.780912 2244 ts_tablet_manager.cc:902] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:22.803421 2170 consensus_queue.cc:1035] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:22.822964 2188 consensus_queue.cc:1035] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:22.830554 1888 consensus_queue.cc:1035] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:22.848693 1814 consensus_queue.cc:1035] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:22.851503 2032 consensus_queue.cc:1035] T d46a094599854588a411f168ee3a262c P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:22.868479 2244 tablet_copy_client.cc:308] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.1:38505
I20240502 08:18:22.869159 27278 tablet_copy_service.cc:140] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Received BeginTabletCopySession request for tablet 20a836dd442943d08a87d61afc785cd1 from peer 84cda20738d2452da166e31fc6da57a6 ({username='slave'} at 127.20.207.6:52973)
I20240502 08:18:22.869371 27278 tablet_copy_service.cc:161] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Beginning new tablet copy session on tablet 20a836dd442943d08a87d61afc785cd1 from peer 84cda20738d2452da166e31fc6da57a6 at {username='slave'} at 127.20.207.6:52973: session id = 84cda20738d2452da166e31fc6da57a6-20a836dd442943d08a87d61afc785cd1
I20240502 08:18:22.873131 27278 tablet_copy_source_session.cc:205] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:22.876629 2244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a836dd442943d08a87d61afc785cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.888679 2244 tablet_copy_client.cc:783] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:22.888921 2244 tablet_copy_client.cc:647] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:22.893756 2244 tablet_copy_client.cc:523] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:22.897686 2244 tablet_bootstrap.cc:492] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:22.973163 2244 tablet_bootstrap.cc:492] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:22.973656 2244 tablet_bootstrap.cc:492] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6: Bootstrap complete.
I20240502 08:18:22.973953 2244 ts_tablet_manager.cc:1367] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.076s user 0.055s sys 0.006s
I20240502 08:18:22.974886 2244 raft_consensus.cc:348] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:22.975229 2244 raft_consensus.cc:729] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: LEARNER
I20240502 08:18:22.975445 2244 consensus_queue.cc:260] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:22.981600 2244 ts_tablet_manager.cc:1398] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:18:22.987458 27278 tablet_copy_service.cc:342] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Request end of tablet copy session 84cda20738d2452da166e31fc6da57a6-20a836dd442943d08a87d61afc785cd1 received from {username='slave'} at 127.20.207.6:52973
I20240502 08:18:22.987592 27278 tablet_copy_service.cc:433] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: ending tablet copy session 84cda20738d2452da166e31fc6da57a6-20a836dd442943d08a87d61afc785cd1 on tablet 20a836dd442943d08a87d61afc785cd1 with peer 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:23.058405 2287 ts_tablet_manager.cc:902] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:23.086753 2287 tablet_copy_client.cc:308] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:23.087431 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet eb5345b931504cfdbb272189b719efa4 from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad ({username='slave'} at 127.20.207.8:38115)
I20240502 08:18:23.087558 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet eb5345b931504cfdbb272189b719efa4 from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad at {username='slave'} at 127.20.207.8:38115: session id = 31a1e33dcdbf42f58ea72ce1c0afb0ad-eb5345b931504cfdbb272189b719efa4
I20240502 08:18:23.090204 27541 tablet_copy_source_session.cc:205] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:23.093961 2287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5345b931504cfdbb272189b719efa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.102942 2287 tablet_copy_client.cc:783] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:23.103193 2287 tablet_copy_client.cc:647] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:23.105024 2287 tablet_copy_client.cc:523] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:23.109279 2287 tablet_bootstrap.cc:492] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:23.150883 30157 raft_consensus.cc:1203] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:18:23.162921 2309 ts_tablet_manager.cc:902] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442: Initiating tablet copy from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:23.196184 2311 ts_tablet_manager.cc:902] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4: Initiating tablet copy from peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:23.211532 2287 tablet_bootstrap.cc:492] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:23.215700 2309 tablet_copy_client.cc:308] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.8:41619
I20240502 08:18:23.216506 30443 tablet_copy_service.cc:140] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Received BeginTabletCopySession request for tablet 1adfeb89dc5341a2b8cf962eb28da45e from peer bf8ac71b15f740a0a928911360773442 ({username='slave'} at 127.20.207.4:55985)
I20240502 08:18:23.216667 30443 tablet_copy_service.cc:161] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Beginning new tablet copy session on tablet 1adfeb89dc5341a2b8cf962eb28da45e from peer bf8ac71b15f740a0a928911360773442 at {username='slave'} at 127.20.207.4:55985: session id = bf8ac71b15f740a0a928911360773442-1adfeb89dc5341a2b8cf962eb28da45e
I20240502 08:18:23.216832 2287 tablet_bootstrap.cc:492] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap complete.
I20240502 08:18:23.217120 2287 ts_tablet_manager.cc:1367] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.108s user 0.050s sys 0.014s
I20240502 08:18:23.217963 2287 raft_consensus.cc:348] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:23.218323 2287 raft_consensus.cc:729] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: LEARNER
I20240502 08:18:23.218551 2287 consensus_queue.cc:260] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:23.219089 2287 ts_tablet_manager.cc:1398] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:23.219604 30443 tablet_copy_source_session.cc:205] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:23.225145 2309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1adfeb89dc5341a2b8cf962eb28da45e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.254973 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-eb5345b931504cfdbb272189b719efa4 received from {username='slave'} at 127.20.207.8:38115
I20240502 08:18:23.255095 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-eb5345b931504cfdbb272189b719efa4 on tablet eb5345b931504cfdbb272189b719efa4 with peer 31a1e33dcdbf42f58ea72ce1c0afb0ad
I20240502 08:18:23.262095 2309 tablet_copy_client.cc:783] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:23.262400 2309 tablet_copy_client.cc:647] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:23.264503 2309 tablet_copy_client.cc:523] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:23.270128 2309 tablet_bootstrap.cc:492] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:23.276994 2311 tablet_copy_client.cc:308] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.6:38899
I20240502 08:18:23.277806 30179 tablet_copy_service.cc:140] P 84cda20738d2452da166e31fc6da57a6: Received BeginTabletCopySession request for tablet b9d6207b6bb3436f996a069ff910a56e from peer 49c5c8f70e494edd9fcda40cc0af6cc4 ({username='slave'} at 127.20.207.7:60051)
I20240502 08:18:23.277978 30179 tablet_copy_service.cc:161] P 84cda20738d2452da166e31fc6da57a6: Beginning new tablet copy session on tablet b9d6207b6bb3436f996a069ff910a56e from peer 49c5c8f70e494edd9fcda40cc0af6cc4 at {username='slave'} at 127.20.207.7:60051: session id = 49c5c8f70e494edd9fcda40cc0af6cc4-b9d6207b6bb3436f996a069ff910a56e
I20240502 08:18:23.278564 2322 ts_tablet_manager.cc:902] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1: Initiating tablet copy from peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:23.280776 30179 tablet_copy_source_session.cc:205] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:23.282076 2311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d6207b6bb3436f996a069ff910a56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.287640 2287 ts_tablet_manager.cc:902] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:23.298290 2309 tablet_bootstrap.cc:492] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:23.298738 2309 tablet_bootstrap.cc:492] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442: Bootstrap complete.
I20240502 08:18:23.299033 2309 ts_tablet_manager.cc:1367] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.003s
I20240502 08:18:23.299933 2309 raft_consensus.cc:348] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:23.300251 2309 raft_consensus.cc:729] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: LEARNER
I20240502 08:18:23.300505 2309 consensus_queue.cc:260] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:23.306953 2309 ts_tablet_manager.cc:1398] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:18:23.319197 30443 tablet_copy_service.cc:342] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Request end of tablet copy session bf8ac71b15f740a0a928911360773442-1adfeb89dc5341a2b8cf962eb28da45e received from {username='slave'} at 127.20.207.4:55985
I20240502 08:18:23.319334 30443 tablet_copy_service.cc:433] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: ending tablet copy session bf8ac71b15f740a0a928911360773442-1adfeb89dc5341a2b8cf962eb28da45e on tablet 1adfeb89dc5341a2b8cf962eb28da45e with peer bf8ac71b15f740a0a928911360773442
I20240502 08:18:23.321775 2311 tablet_copy_client.cc:783] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:23.322046 2311 tablet_copy_client.cc:647] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:23.330852 2311 tablet_copy_client.cc:523] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:23.335271 2287 tablet_copy_client.cc:308] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:23.336278 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet c2776f857a5f47e9b8022639dc54757d from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad ({username='slave'} at 127.20.207.8:38115)
I20240502 08:18:23.336422 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet c2776f857a5f47e9b8022639dc54757d from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad at {username='slave'} at 127.20.207.8:38115: session id = 31a1e33dcdbf42f58ea72ce1c0afb0ad-c2776f857a5f47e9b8022639dc54757d
I20240502 08:18:23.339118 27541 tablet_copy_source_session.cc:205] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:23.348750 2322 tablet_copy_client.cc:308] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.5:37815
I20240502 08:18:23.349584 30048 tablet_copy_service.cc:140] P 1f0bf4bc3c5146539146b825b048bff3: Received BeginTabletCopySession request for tablet 26545a868bd6415c9c9db4869b2841c7 from peer 83e171a5e16c4008923f1d43fc5ce7b1 ({username='slave'} at 127.20.207.2:49697)
I20240502 08:18:23.349731 30048 tablet_copy_service.cc:161] P 1f0bf4bc3c5146539146b825b048bff3: Beginning new tablet copy session on tablet 26545a868bd6415c9c9db4869b2841c7 from peer 83e171a5e16c4008923f1d43fc5ce7b1 at {username='slave'} at 127.20.207.2:49697: session id = 83e171a5e16c4008923f1d43fc5ce7b1-26545a868bd6415c9c9db4869b2841c7
I20240502 08:18:23.352422 30048 tablet_copy_source_session.cc:205] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:23.362378 2322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26545a868bd6415c9c9db4869b2841c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.373482 2322 tablet_copy_client.cc:783] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:23.373653 2322 tablet_copy_client.cc:647] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:23.383304 2287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2776f857a5f47e9b8022639dc54757d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.395366 2322 tablet_copy_client.cc:523] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:23.402701 2322 tablet_bootstrap.cc:492] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:23.417901 2322 tablet_bootstrap.cc:492] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:23.418465 2322 tablet_bootstrap.cc:492] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap complete.
I20240502 08:18:23.418730 2322 ts_tablet_manager.cc:1367] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20240502 08:18:23.419360 2322 raft_consensus.cc:348] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:23.419593 2322 raft_consensus.cc:729] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: LEARNER
I20240502 08:18:23.419752 2322 consensus_queue.cc:260] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:23.420822 2322 ts_tablet_manager.cc:1398] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:23.431455 30048 tablet_copy_service.cc:342] P 1f0bf4bc3c5146539146b825b048bff3: Request end of tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-26545a868bd6415c9c9db4869b2841c7 received from {username='slave'} at 127.20.207.2:49697
I20240502 08:18:23.431661 30048 tablet_copy_service.cc:433] P 1f0bf4bc3c5146539146b825b048bff3: ending tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-26545a868bd6415c9c9db4869b2841c7 on tablet 26545a868bd6415c9c9db4869b2841c7 with peer 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:23.432852 2311 tablet_bootstrap.cc:492] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:23.443303 2287 tablet_copy_client.cc:783] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:23.443617 2287 tablet_copy_client.cc:647] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:23.445297 2287 tablet_copy_client.cc:523] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:23.449278 2287 tablet_bootstrap.cc:492] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:23.460031 2311 tablet_bootstrap.cc:492] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:23.460634 2311 tablet_bootstrap.cc:492] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap complete.
I20240502 08:18:23.461443 2311 ts_tablet_manager.cc:1367] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.000s
I20240502 08:18:23.463166 2311 raft_consensus.cc:348] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:23.464181 2311 raft_consensus.cc:729] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: LEARNER
I20240502 08:18:23.464460 2311 consensus_queue.cc:260] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:23.466233 2311 ts_tablet_manager.cc:1398] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:23.467061 30179 tablet_copy_service.cc:342] P 84cda20738d2452da166e31fc6da57a6: Request end of tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-b9d6207b6bb3436f996a069ff910a56e received from {username='slave'} at 127.20.207.7:60051
I20240502 08:18:23.467201 30179 tablet_copy_service.cc:433] P 84cda20738d2452da166e31fc6da57a6: ending tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-b9d6207b6bb3436f996a069ff910a56e on tablet b9d6207b6bb3436f996a069ff910a56e with peer 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:23.472746 30422 raft_consensus.cc:1203] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:18:23.520503 27257 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "15a333c65455473fbdc26d1ec137aebc"
dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47666
I20240502 08:18:23.520689 27257 raft_consensus.cc:595] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:18:23.541299 2287 tablet_bootstrap.cc:492] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:23.541795 2287 tablet_bootstrap.cc:492] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap complete.
I20240502 08:18:23.542085 2287 ts_tablet_manager.cc:1367] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.093s user 0.051s sys 0.006s
I20240502 08:18:23.542922 2287 raft_consensus.cc:348] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:23.543136 2287 raft_consensus.cc:729] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: LEARNER
I20240502 08:18:23.543304 2287 consensus_queue.cc:260] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:23.543959 2287 ts_tablet_manager.cc:1398] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:23.545213 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-c2776f857a5f47e9b8022639dc54757d received from {username='slave'} at 127.20.207.8:38115
I20240502 08:18:23.545318 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-c2776f857a5f47e9b8022639dc54757d on tablet c2776f857a5f47e9b8022639dc54757d with peer 31a1e33dcdbf42f58ea72ce1c0afb0ad
I20240502 08:18:23.551362 2056 raft_consensus.cc:1050] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: attempting to promote NON_VOTER 84cda20738d2452da166e31fc6da57a6 to VOTER
I20240502 08:18:23.552119 2056 consensus_queue.cc:237] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:23.555159 30423 raft_consensus.cc:1261] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:23.555258 30157 raft_consensus.cc:1261] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:23.555289 27652 raft_consensus.cc:1261] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:23.555856 1787 consensus_queue.cc:1035] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:23.556424 1787 consensus_queue.cc:1035] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:23.556938 1787 consensus_queue.cc:1035] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:23.561254 27518 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "09aa1b5d99114b3095be08c9547fbb54"
dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57822
I20240502 08:18:23.561446 27518 raft_consensus.cc:595] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:18:23.561944 1787 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:23.562707 30157 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:23.570525 27112 catalog_manager.cc:5484] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:23.583009 27652 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:23.584937 30423 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:23.588078 30290 raft_consensus.cc:1203] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:23.597720 1551 raft_consensus.cc:979] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: : Instructing follower 31a1e33dcdbf42f58ea72ce1c0afb0ad to start an election
I20240502 08:18:23.597841 1551 raft_consensus.cc:1067] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Signalling peer 31a1e33dcdbf42f58ea72ce1c0afb0ad to start an election
I20240502 08:18:23.599839 30422 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "15a333c65455473fbdc26d1ec137aebc"
dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
from {username='slave'} at 127.20.207.1:41297
I20240502 08:18:23.600069 30422 raft_consensus.cc:482] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:23.600162 30422 raft_consensus.cc:3037] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:23.605794 27257 consensus_queue.cc:237] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:23.607733 30157 raft_consensus.cc:1261] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:23.610328 30422 raft_consensus.cc:504] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:23.611315 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a333c65455473fbdc26d1ec137aebc" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:23.611440 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a333c65455473fbdc26d1ec137aebc" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:23.611388 1787 consensus_queue.cc:1035] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:23.611588 27257 raft_consensus.cc:3032] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:23.611677 27257 raft_consensus.cc:729] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:18:23.611907 27257 consensus_queue.cc:260] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:23.612289 27257 raft_consensus.cc:3037] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:23.612478 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a333c65455473fbdc26d1ec137aebc" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:23.612682 30290 raft_consensus.cc:3037] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:23.613094 27652 raft_consensus.cc:3037] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:23.614081 30422 leader_election.cc:288] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:23.616027 30290 raft_consensus.cc:2454] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 2.
I20240502 08:18:23.616226 1787 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:23.617595 27652 raft_consensus.cc:2454] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 2.
I20240502 08:18:23.618047 30358 leader_election.cc:302] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:23.618346 2353 raft_consensus.cc:2781] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:23.618530 2353 raft_consensus.cc:686] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:18:23.618804 2353 consensus_queue.cc:237] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:23.621634 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 20a836dd442943d08a87d61afc785cd1 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:23.622303 2351 raft_consensus.cc:979] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59: : Instructing follower 1f0bf4bc3c5146539146b825b048bff3 to start an election
I20240502 08:18:23.618387 30423 raft_consensus.cc:1261] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:23.622431 2351 raft_consensus.cc:1067] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Signalling peer 1f0bf4bc3c5146539146b825b048bff3 to start an election
I20240502 08:18:23.622532 30157 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:23.623087 30028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "09aa1b5d99114b3095be08c9547fbb54"
dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
from {username='slave'} at 127.20.207.3:54281
I20240502 08:18:23.623270 30028 raft_consensus.cc:482] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:23.623361 30028 raft_consensus.cc:3037] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:23.625916 30028 raft_consensus.cc:504] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:23.626945 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09aa1b5d99114b3095be08c9547fbb54" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:23.627045 1551 consensus_queue.cc:1035] T 20a836dd442943d08a87d61afc785cd1 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:23.627154 27255 raft_consensus.cc:3037] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:23.629853 27255 raft_consensus.cc:2454] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 2.
I20240502 08:18:23.633872 27112 catalog_manager.cc:5484] T 20a836dd442943d08a87d61afc785cd1 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 8 to 9, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:23.636343 27113 catalog_manager.cc:5484] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 1 to 2, leader changed from 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 2 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:23.640684 27629 tablet_service.cc:1514] Processing DeleteTablet for tablet 20a836dd442943d08a87d61afc785cd1 with delete_type TABLET_DATA_TOMBSTONED (TS bf8ac71b15f740a0a928911360773442 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44788
I20240502 08:18:23.642603 2387 tablet_replica.cc:330] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:18:23.642827 2387 raft_consensus.cc:2229] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:23.643081 2387 raft_consensus.cc:2258] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:23.643050 27518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09aa1b5d99114b3095be08c9547fbb54" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:18:23.643208 27518 raft_consensus.cc:3032] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:23.643262 27518 raft_consensus.cc:729] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:18:23.643477 27518 consensus_queue.cc:260] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:23.643723 30028 leader_election.cc:288] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:23.643826 27518 raft_consensus.cc:3037] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:23.645354 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09aa1b5d99114b3095be08c9547fbb54" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:23.645576 30290 raft_consensus.cc:3037] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:23.630546 27257 raft_consensus.cc:2454] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 2.
I20240502 08:18:23.647262 27518 raft_consensus.cc:2454] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 2.
I20240502 08:18:23.647689 29964 leader_election.cc:302] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 1f0bf4bc3c5146539146b825b048bff3, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:23.647989 2364 raft_consensus.cc:2781] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:23.648149 2364 raft_consensus.cc:686] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:18:23.648617 2364 consensus_queue.cc:237] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:23.655673 27113 catalog_manager.cc:5484] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 1 to 2, leader changed from b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 2 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:23.660575 30290 raft_consensus.cc:2454] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 2.
I20240502 08:18:23.660851 2387 ts_tablet_manager.cc:1875] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:23.665095 30423 raft_consensus.cc:2932] T 20a836dd442943d08a87d61afc785cd1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:23.668047 2387 ts_tablet_manager.cc:1888] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:18:23.668154 2387 log.cc:1196] T 20a836dd442943d08a87d61afc785cd1 P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/20a836dd442943d08a87d61afc785cd1
I20240502 08:18:23.668838 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 20a836dd442943d08a87d61afc785cd1 (table rebalance_test_table_19 [id=7f9af7f2cb49474eb721698045f06a63]) successfully deleted
I20240502 08:18:23.672607 27652 raft_consensus.cc:1203] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:23.680970 27389 raft_consensus.cc:1203] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:18:23.719154 27652 consensus_queue.cc:237] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:23.723323 27518 raft_consensus.cc:1261] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:18:23.723766 27587 consensus_peers.cc:485] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 -> Peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Couldn't send request to peer 84cda20738d2452da166e31fc6da57a6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a31709aa6b2b4ff784114fb48cbc8f2a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:23.724121 30423 raft_consensus.cc:1261] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:23.724623 2267 consensus_queue.cc:1035] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:23.725354 2267 consensus_queue.cc:1035] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:23.730703 2267 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:23.734766 27516 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:23.740401 30423 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:23.743602 27113 catalog_manager.cc:5484] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:23.867756 30423 raft_consensus.cc:1203] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:18:23.949206 2202 raft_consensus.cc:1050] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad to VOTER
I20240502 08:18:23.950054 2202 consensus_queue.cc:237] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:23.963163 27257 raft_consensus.cc:1261] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:23.964080 1729 consensus_queue.cc:1035] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:23.965168 30423 raft_consensus.cc:1261] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:23.965706 2351 consensus_queue.cc:1035] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:23.967809 30157 raft_consensus.cc:1261] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:23.968695 2351 consensus_queue.cc:1035] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:24.012027 2351 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:24.014881 30156 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:24.028241 30423 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:24.029100 27112 catalog_manager.cc:5484] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 7 to 8, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:24.034215 27516 consensus_queue.cc:237] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:24.037705 27649 raft_consensus.cc:1261] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:24.037966 30159 raft_consensus.cc:1261] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:24.038641 2353 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:24.038672 1729 consensus_queue.cc:1035] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:24.041797 1814 raft_consensus.cc:1050] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6: attempting to promote NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 to VOTER
I20240502 08:18:24.042438 30423 raft_consensus.cc:1261] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:24.042553 1814 consensus_queue.cc:237] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:24.042847 1729 consensus_queue.cc:1035] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:24.044845 1729 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:24.046809 30159 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:24.047909 30422 raft_consensus.cc:2932] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:24.055101 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb5345b931504cfdbb272189b719efa4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:24.059159 27112 catalog_manager.cc:5484] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 8 to 9, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:24.062848 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet eb5345b931504cfdbb272189b719efa4 with delete_type TABLET_DATA_TOMBSTONED (TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39166
I20240502 08:18:24.078825 2353 raft_consensus.cc:1050] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: attempting to promote NON_VOTER bf8ac71b15f740a0a928911360773442 to VOTER
I20240502 08:18:24.079449 2448 tablet_replica.cc:330] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:18:24.079735 2448 raft_consensus.cc:2229] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:24.079654 2353 consensus_queue.cc:237] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:24.080062 2448 pending_rounds.cc:62] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Trying to abort 1 pending ops.
I20240502 08:18:24.080181 2448 pending_rounds.cc:68] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023156854577168384 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "eb5345b931504cfdbb272189b719efa4" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } } }
I20240502 08:18:24.080585 2448 raft_consensus.cc:2894] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240502 08:18:24.080719 2448 raft_consensus.cc:2866] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 08:18:24.080854 2448 raft_consensus.cc:2258] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:24.081823 27389 raft_consensus.cc:1261] T b9d6207b6bb3436f996a069ff910a56e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:24.082278 30290 raft_consensus.cc:1261] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:24.082409 1814 consensus_queue.cc:1035] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:24.082868 2448 ts_tablet_manager.cc:1875] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:24.083236 30423 raft_consensus.cc:1261] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:24.083863 30290 raft_consensus.cc:1261] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:24.084417 2281 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:24.086814 2033 consensus_queue.cc:1035] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:24.087524 2033 consensus_queue.cc:1035] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:24.089087 2448 ts_tablet_manager.cc:1888] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:18:24.089198 2448 log.cc:1196] T eb5345b931504cfdbb272189b719efa4 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/eb5345b931504cfdbb272189b719efa4
I20240502 08:18:24.089982 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet eb5345b931504cfdbb272189b719efa4 (table rebalance_test_table_15 [id=f2de4023ae664751b94f83ff4434cd92]) successfully deleted
I20240502 08:18:24.090994 27649 raft_consensus.cc:1261] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:24.091017 30159 raft_consensus.cc:1261] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:24.091272 27516 raft_consensus.cc:1261] T 1adfeb89dc5341a2b8cf962eb28da45e P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:24.092157 2353 consensus_queue.cc:1035] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:18:24.092798 1755 consensus_queue.cc:1035] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:24.093307 1755 consensus_queue.cc:1035] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:24.108639 27257 raft_consensus.cc:1261] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:24.108539 2033 raft_consensus.cc:2932] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:24.110467 2453 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:24.117045 30289 raft_consensus.cc:2932] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:24.119266 27389 raft_consensus.cc:2932] T b9d6207b6bb3436f996a069ff910a56e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:24.127636 27115 catalog_manager.cc:5484] T b9d6207b6bb3436f996a069ff910a56e P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:24.140324 30288 raft_consensus.cc:1261] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:24.141139 2353 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:24.158594 2462 ts_tablet_manager.cc:902] T a31709aa6b2b4ff784114fb48cbc8f2a P 84cda20738d2452da166e31fc6da57a6: Initiating tablet copy from peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:24.158898 27516 raft_consensus.cc:1261] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 2 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:24.159555 2281 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:24.172688 2462 tablet_copy_client.cc:308] T a31709aa6b2b4ff784114fb48cbc8f2a P 84cda20738d2452da166e31fc6da57a6: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.4:33309
I20240502 08:18:24.174391 27671 tablet_copy_service.cc:140] P bf8ac71b15f740a0a928911360773442: Received BeginTabletCopySession request for tablet a31709aa6b2b4ff784114fb48cbc8f2a from peer 84cda20738d2452da166e31fc6da57a6 ({username='slave'} at 127.20.207.6:47641)
I20240502 08:18:24.174531 27671 tablet_copy_service.cc:161] P bf8ac71b15f740a0a928911360773442: Beginning new tablet copy session on tablet a31709aa6b2b4ff784114fb48cbc8f2a from peer 84cda20738d2452da166e31fc6da57a6 at {username='slave'} at 127.20.207.6:47641: session id = 84cda20738d2452da166e31fc6da57a6-a31709aa6b2b4ff784114fb48cbc8f2a
I20240502 08:18:24.177227 27671 tablet_copy_source_session.cc:205] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:24.189386 2462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a31709aa6b2b4ff784114fb48cbc8f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:24.191443 30423 raft_consensus.cc:2932] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:24.201558 2453 raft_consensus.cc:2932] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:24.219511 27650 raft_consensus.cc:2932] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:24.220427 27518 raft_consensus.cc:2932] T 1adfeb89dc5341a2b8cf962eb28da45e P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:24.225896 27257 raft_consensus.cc:1261] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:24.231788 2281 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:24.232486 2281 raft_consensus.cc:1050] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3: attempting to promote NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 to VOTER
I20240502 08:18:24.233359 2281 consensus_queue.cc:237] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:24.233621 2462 tablet_copy_client.cc:783] T a31709aa6b2b4ff784114fb48cbc8f2a P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:24.234282 27110 catalog_manager.cc:5484] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:24.236078 2462 tablet_copy_client.cc:647] T a31709aa6b2b4ff784114fb48cbc8f2a P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:24.240274 2462 tablet_copy_client.cc:523] T a31709aa6b2b4ff784114fb48cbc8f2a P 84cda20738d2452da166e31fc6da57a6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:24.240362 30157 raft_consensus.cc:2932] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:24.246647 2462 tablet_bootstrap.cc:492] T a31709aa6b2b4ff784114fb48cbc8f2a P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:24.255482 30156 consensus_queue.cc:237] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:24.257702 2442 raft_consensus.cc:1050] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad to VOTER
I20240502 08:18:24.258500 2442 consensus_queue.cc:237] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:24.258690 27389 raft_consensus.cc:1261] T b9d6207b6bb3436f996a069ff910a56e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:24.259318 30288 raft_consensus.cc:1261] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:24.259434 1814 consensus_queue.cc:1035] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:24.263005 1814 consensus_queue.cc:1035] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:24.264398 2033 raft_consensus.cc:2932] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:24.266490 30288 raft_consensus.cc:2932] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:24.267843 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b9d6207b6bb3436f996a069ff910a56e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:24.269834 27389 raft_consensus.cc:2932] T b9d6207b6bb3436f996a069ff910a56e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:24.274497 27114 catalog_manager.cc:5484] T b9d6207b6bb3436f996a069ff910a56e P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:24.286623 30419 raft_consensus.cc:1261] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:24.287292 1729 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:24.287804 30423 consensus_queue.cc:237] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:24.291642 27650 raft_consensus.cc:1261] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:24.292140 2473 consensus_queue.cc:1035] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:24.294900 1755 raft_consensus.cc:2932] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:24.296952 30288 raft_consensus.cc:1261] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:24.297606 1729 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:24.300449 27516 raft_consensus.cc:1261] T 1adfeb89dc5341a2b8cf962eb28da45e P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:24.300936 2353 consensus_queue.cc:1035] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:24.303529 27516 raft_consensus.cc:2932] T 1adfeb89dc5341a2b8cf962eb28da45e P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:24.306429 27649 raft_consensus.cc:2932] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:24.308965 27650 raft_consensus.cc:1261] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:24.314146 2442 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:24.325605 27389 raft_consensus.cc:1261] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:24.326315 30159 raft_consensus.cc:1261] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:24.326862 27257 raft_consensus.cc:1261] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:24.327026 2364 consensus_queue.cc:1035] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:24.329314 2474 consensus_queue.cc:1035] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:24.330964 2474 consensus_queue.cc:1035] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:24.332680 30401 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d6207b6bb3436f996a069ff910a56e with delete_type TABLET_DATA_TOMBSTONED (TS 31a1e33dcdbf42f58ea72ce1c0afb0ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:24.332872 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1adfeb89dc5341a2b8cf962eb28da45e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:24.334266 27108 catalog_manager.cc:5484] T 1adfeb89dc5341a2b8cf962eb28da45e P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:24.341215 30139 tablet_service.cc:1514] Processing DeleteTablet for tablet 1adfeb89dc5341a2b8cf962eb28da45e with delete_type TABLET_DATA_TOMBSTONED (TS 84cda20738d2452da166e31fc6da57a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:24.360698 2202 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:24.369299 30289 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:24.370931 27650 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:24.371170 2474 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:24.373355 30419 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:24.375157 27257 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:24.376356 30156 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:24.381443 27114 catalog_manager.cc:5484] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 7 to 8, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:24.400461 27114 catalog_manager.cc:5484] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 5 to 6, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:24.411976 2489 tablet_replica.cc:330] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:24.412214 2489 raft_consensus.cc:2229] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:24.417835 30026 consensus_queue.cc:237] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:24.420864 27257 raft_consensus.cc:1261] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:24.421478 2476 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:24.422808 30289 raft_consensus.cc:1261] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:24.422668 27389 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:24.423409 2476 consensus_queue.cc:1035] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:24.425122 2476 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:24.430550 27255 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:24.421980 2489 raft_consensus.cc:2258] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:24.438663 2488 tablet_replica.cc:330] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:24.439553 2489 ts_tablet_manager.cc:1875] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:24.439611 30028 consensus_queue.cc:237] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:24.442404 27255 raft_consensus.cc:1261] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:24.442871 2281 consensus_queue.cc:1035] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:24.442649 30289 raft_consensus.cc:2932] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:24.445238 2281 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:24.450076 2488 raft_consensus.cc:2229] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:24.450611 27255 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:24.451344 2488 raft_consensus.cc:2258] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:24.452754 2488 ts_tablet_manager.cc:1875] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:24.460281 27389 raft_consensus.cc:1261] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:24.460994 2476 consensus_queue.cc:1035] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:24.462795 27388 raft_consensus.cc:2932] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:24.463977 27114 catalog_manager.cc:5484] T 09aa1b5d99114b3095be08c9547fbb54 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 8 to 10, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 2 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:24.468235 27498 tablet_service.cc:1514] Processing DeleteTablet for tablet 09aa1b5d99114b3095be08c9547fbb54 with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:24.473459 2489 ts_tablet_manager.cc:1888] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:24.473593 2489 log.cc:1196] T 1adfeb89dc5341a2b8cf962eb28da45e P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/1adfeb89dc5341a2b8cf962eb28da45e
I20240502 08:18:24.476569 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 26545a868bd6415c9c9db4869b2841c7 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:24.476840 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 09aa1b5d99114b3095be08c9547fbb54 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:24.477502 27114 catalog_manager.cc:5484] T 26545a868bd6415c9c9db4869b2841c7 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 6 to 7, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:24.481907 30139 tablet_service.cc:1514] Processing DeleteTablet for tablet 26545a868bd6415c9c9db4869b2841c7 with delete_type TABLET_DATA_TOMBSTONED (TS 84cda20738d2452da166e31fc6da57a6 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:24.483337 2495 tablet_replica.cc:330] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:24.488956 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet 1adfeb89dc5341a2b8cf962eb28da45e (table rebalancer_extra_table_32 [id=3c7f8803df8a4ae885409a0c0a114f1f]) successfully deleted
I20240502 08:18:24.492674 2489 tablet_replica.cc:330] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:24.493325 2489 raft_consensus.cc:2229] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:24.494122 2489 raft_consensus.cc:2258] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:24.494321 2488 ts_tablet_manager.cc:1888] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:24.494598 2488 log.cc:1196] T b9d6207b6bb3436f996a069ff910a56e P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/b9d6207b6bb3436f996a069ff910a56e
I20240502 08:18:24.498610 2489 ts_tablet_manager.cc:1875] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:24.508174 27238 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:24.508641 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet b9d6207b6bb3436f996a069ff910a56e (table rebalancer_extra_table_16 [id=98426515f8594fe1951ef5a81f079c16]) successfully deleted
I20240502 08:18:24.510077 2495 raft_consensus.cc:2229] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:24.510251 2495 raft_consensus.cc:2258] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:24.513350 2495 ts_tablet_manager.cc:1875] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:24.516810 2489 ts_tablet_manager.cc:1888] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:18:24.517005 2489 log.cc:1196] T 26545a868bd6415c9c9db4869b2841c7 P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/26545a868bd6415c9c9db4869b2841c7
I20240502 08:18:24.524541 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet 26545a868bd6415c9c9db4869b2841c7 (table rebalancer_extra_table_10 [id=9df4632350e6444089d25e0f410be77a]) successfully deleted
I20240502 08:18:24.536965 2495 ts_tablet_manager.cc:1888] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 08:18:24.539431 2495 log.cc:1196] T 09aa1b5d99114b3095be08c9547fbb54 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/09aa1b5d99114b3095be08c9547fbb54
I20240502 08:18:24.540365 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet 09aa1b5d99114b3095be08c9547fbb54 (table rebalance_test_table_9 [id=a65036830fa64a22b9f4efd6d5bd4f9f]) successfully deleted
I20240502 08:18:24.561580 30401 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 08:18:24.580049 27366 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:18:24.587749 30007 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:24.590642 30270 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:18:24.647137 2462 tablet_bootstrap.cc:492] T a31709aa6b2b4ff784114fb48cbc8f2a P 84cda20738d2452da166e31fc6da57a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:24.647919 2462 tablet_bootstrap.cc:492] T a31709aa6b2b4ff784114fb48cbc8f2a P 84cda20738d2452da166e31fc6da57a6: Bootstrap complete.
I20240502 08:18:24.656981 2462 ts_tablet_manager.cc:1367] T a31709aa6b2b4ff784114fb48cbc8f2a P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.410s user 0.066s sys 0.014s
I20240502 08:18:24.662632 2462 raft_consensus.cc:348] T a31709aa6b2b4ff784114fb48cbc8f2a P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:24.663414 2462 raft_consensus.cc:729] T a31709aa6b2b4ff784114fb48cbc8f2a P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: LEARNER
I20240502 08:18:24.663842 2462 consensus_queue.cc:260] T a31709aa6b2b4ff784114fb48cbc8f2a P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:24.664851 2462 ts_tablet_manager.cc:1398] T a31709aa6b2b4ff784114fb48cbc8f2a P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:18:24.666620 27671 tablet_copy_service.cc:342] P bf8ac71b15f740a0a928911360773442: Request end of tablet copy session 84cda20738d2452da166e31fc6da57a6-a31709aa6b2b4ff784114fb48cbc8f2a received from {username='slave'} at 127.20.207.6:47641
I20240502 08:18:24.666754 27671 tablet_copy_service.cc:433] P bf8ac71b15f740a0a928911360773442: ending tablet copy session 84cda20738d2452da166e31fc6da57a6-a31709aa6b2b4ff784114fb48cbc8f2a on tablet a31709aa6b2b4ff784114fb48cbc8f2a with peer 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:24.691324 27629 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:18:24.692165 27498 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:24.701745 30423 consensus_queue.cc:237] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:24.703745 30288 raft_consensus.cc:1261] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:24.704282 2353 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:24.708493 2453 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:24.709084 30289 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:24.718168 27649 raft_consensus.cc:1261] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:24.719010 1755 consensus_queue.cc:1035] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:24.723031 27649 raft_consensus.cc:2932] T 15a333c65455473fbdc26d1ec137aebc P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:24.723870 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 15a333c65455473fbdc26d1ec137aebc with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:24.728029 27114 catalog_manager.cc:5484] T 15a333c65455473fbdc26d1ec137aebc P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 8 to 10, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New cstate: current_term: 2 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:24.733534 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet 15a333c65455473fbdc26d1ec137aebc with delete_type TABLET_DATA_TOMBSTONED (TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:39166
I20240502 08:18:24.740368 30139 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:24.768098 2512 tablet_replica.cc:330] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:18:24.771939 2512 raft_consensus.cc:2229] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:24.772231 2512 raft_consensus.cc:2258] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:24.773885 2512 ts_tablet_manager.cc:1875] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:24.798763 2512 ts_tablet_manager.cc:1888] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 08:18:24.798890 2512 log.cc:1196] T 15a333c65455473fbdc26d1ec137aebc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/15a333c65455473fbdc26d1ec137aebc
I20240502 08:18:24.799609 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet 15a333c65455473fbdc26d1ec137aebc (table rebalance_test_table_1 [id=33e20eb07c524d63b8e99e7957467d8b]) successfully deleted
I20240502 08:18:24.974243 30159 consensus_queue.cc:237] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:24.976951 30289 raft_consensus.cc:1261] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:24.976989 27255 raft_consensus.cc:1261] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:24.977538 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9edac369e0ae4c3c990bbb8dd845b27c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:24.978066 2457 consensus_queue.cc:1035] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:24.978724 2457 consensus_queue.cc:1035] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20240502 08:18:24.990077 27255 raft_consensus.cc:2932] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:24.991369 30289 raft_consensus.cc:2932] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:24.989495 2520 raft_consensus.cc:2932] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:24.997545 27112 catalog_manager.cc:5484] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:25.017951 27518 consensus_queue.cc:237] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:25.020490 27388 raft_consensus.cc:1261] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:18:25.020630 27455 consensus_peers.cc:485] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 -> Peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): Couldn't send request to peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e0313e85d444e098bfa35be89a3ad97. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:25.020970 2529 consensus_queue.cc:1035] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:25.021442 30026 raft_consensus.cc:1261] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:25.021929 2529 consensus_queue.cc:1035] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:25.024516 1729 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:25.025007 27388 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:25.030354 27112 catalog_manager.cc:5484] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index 6 to 7, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:25.033172 30026 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:25.045668 27518 consensus_queue.cc:237] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:25.047562 27255 raft_consensus.cc:1261] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:25.047748 30289 raft_consensus.cc:1261] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 08:18:25.047986 27456 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 -> Peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Couldn't send request to peer bf8ac71b15f740a0a928911360773442. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de4b822c4aca4f7f92b73cc9279ddec5. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:25.048236 2529 consensus_queue.cc:1035] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:25.048453 1729 consensus_queue.cc:1035] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:25.050824 2529 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:25.051600 30288 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:25.055670 27255 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:25.065135 27112 catalog_manager.cc:5484] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 4 to 5, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:25.080631 30159 raft_consensus.cc:1203] T a31709aa6b2b4ff784114fb48cbc8f2a P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:18:25.090638 30026 consensus_queue.cc:237] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:25.093103 30419 raft_consensus.cc:1261] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:25.093127 27255 raft_consensus.cc:1261] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:25.093438 29963 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e9b191ce9234c028b990efd74a59e2e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:25.093647 2281 consensus_queue.cc:1035] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:25.093904 2476 consensus_queue.cc:1035] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:25.096634 2281 raft_consensus.cc:2932] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:25.098053 27255 raft_consensus.cc:2932] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:25.098325 30423 raft_consensus.cc:2932] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:25.107492 27115 catalog_manager.cc:5484] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:25.128242 30423 consensus_queue.cc:237] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:25.130230 27518 raft_consensus.cc:1261] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:25.130493 27389 raft_consensus.cc:1261] T d3a3473a93f14baaac48b9b7f47ac2a1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:25.130538 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3a3473a93f14baaac48b9b7f47ac2a1. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:25.130733 2353 consensus_queue.cc:1035] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:25.131029 2453 consensus_queue.cc:1035] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:25.133564 2453 raft_consensus.cc:2932] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:25.136739 27518 raft_consensus.cc:2932] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:25.137295 27389 raft_consensus.cc:2932] T d3a3473a93f14baaac48b9b7f47ac2a1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:25.149852 27112 catalog_manager.cc:5484] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:25.154821 30026 consensus_queue.cc:237] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:25.157660 27388 raft_consensus.cc:1261] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:25.158361 2550 consensus_queue.cc:1035] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:25.159373 29963 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): Couldn't send request to peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3a55662c6e346f0b73df334be4b14e3. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:25.160878 30288 raft_consensus.cc:1261] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:25.161535 2550 consensus_queue.cc:1035] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:25.163774 2281 raft_consensus.cc:2932] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:25.164990 27388 raft_consensus.cc:2932] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:25.170500 27114 catalog_manager.cc:5484] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:25.187358 30423 consensus_queue.cc:237] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:25.189476 30159 raft_consensus.cc:1261] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:25.189659 30288 raft_consensus.cc:1261] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:25.189893 30358 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): Couldn't send request to peer bf8ac71b15f740a0a928911360773442. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73548c1dd45f4c31bf3caf89b6d9ebac. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:25.190286 2453 consensus_queue.cc:1035] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:25.190747 2453 consensus_queue.cc:1035] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:25.191583 30288 raft_consensus.cc:2932] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:25.214828 2453 raft_consensus.cc:2932] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:25.215869 30289 raft_consensus.cc:2932] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:25.219344 30159 raft_consensus.cc:2932] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } } }
I20240502 08:18:25.228471 27115 catalog_manager.cc:5484] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index -1 to 2, NON_VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) added. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:25.263283 30289 consensus_queue.cc:237] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:25.265503 30423 raft_consensus.cc:1261] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:25.266592 30159 raft_consensus.cc:1261] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:25.267020 30223 consensus_peers.cc:485] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 -> Peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): Couldn't send request to peer 1f0bf4bc3c5146539146b825b048bff3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1f5650965e841ad98d857ca32299037. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:25.267433 2464 consensus_queue.cc:1035] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:25.267740 2465 consensus_queue.cc:1035] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:25.273520 2464 raft_consensus.cc:2932] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:25.274755 30159 raft_consensus.cc:2932] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:25.281221 27114 catalog_manager.cc:5484] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index -1 to 2, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:25.282372 30423 raft_consensus.cc:2932] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } } }
I20240502 08:18:25.320760 30289 consensus_queue.cc:237] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:25.322774 27652 raft_consensus.cc:1261] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:25.322849 30423 raft_consensus.cc:1261] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:25.323297 2588 consensus_queue.cc:1035] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:25.323575 30226 consensus_peers.cc:485] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca8d1416a54344feae081ed7b19dd1de. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:25.323836 2588 consensus_queue.cc:1035] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:25.325795 2587 raft_consensus.cc:2932] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:25.326493 27652 raft_consensus.cc:2932] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:25.328857 30423 raft_consensus.cc:2932] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:25.332010 27115 catalog_manager.cc:5484] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:25.485662 2640 ts_tablet_manager.cc:902] T 9edac369e0ae4c3c990bbb8dd845b27c P 83e171a5e16c4008923f1d43fc5ce7b1: Initiating tablet copy from peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:25.486878 2640 tablet_copy_client.cc:308] T 9edac369e0ae4c3c990bbb8dd845b27c P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.6:38899
I20240502 08:18:25.487488 30179 tablet_copy_service.cc:140] P 84cda20738d2452da166e31fc6da57a6: Received BeginTabletCopySession request for tablet 9edac369e0ae4c3c990bbb8dd845b27c from peer 83e171a5e16c4008923f1d43fc5ce7b1 ({username='slave'} at 127.20.207.2:56535)
I20240502 08:18:25.487684 30179 tablet_copy_service.cc:161] P 84cda20738d2452da166e31fc6da57a6: Beginning new tablet copy session on tablet 9edac369e0ae4c3c990bbb8dd845b27c from peer 83e171a5e16c4008923f1d43fc5ce7b1 at {username='slave'} at 127.20.207.2:56535: session id = 83e171a5e16c4008923f1d43fc5ce7b1-9edac369e0ae4c3c990bbb8dd845b27c
I20240502 08:18:25.491691 30179 tablet_copy_source_session.cc:205] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:25.493178 2640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9edac369e0ae4c3c990bbb8dd845b27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.499513 2640 tablet_copy_client.cc:783] T 9edac369e0ae4c3c990bbb8dd845b27c P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:25.499765 2640 tablet_copy_client.cc:647] T 9edac369e0ae4c3c990bbb8dd845b27c P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:25.501071 2640 tablet_copy_client.cc:523] T 9edac369e0ae4c3c990bbb8dd845b27c P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:25.503888 2640 tablet_bootstrap.cc:492] T 9edac369e0ae4c3c990bbb8dd845b27c P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:25.509835 2640 tablet_bootstrap.cc:492] T 9edac369e0ae4c3c990bbb8dd845b27c P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:25.510232 2640 tablet_bootstrap.cc:492] T 9edac369e0ae4c3c990bbb8dd845b27c P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap complete.
I20240502 08:18:25.510462 2640 ts_tablet_manager.cc:1367] T 9edac369e0ae4c3c990bbb8dd845b27c P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20240502 08:18:25.511106 2640 raft_consensus.cc:348] T 9edac369e0ae4c3c990bbb8dd845b27c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:25.511325 2640 raft_consensus.cc:729] T 9edac369e0ae4c3c990bbb8dd845b27c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: LEARNER
I20240502 08:18:25.511492 2640 consensus_queue.cc:260] T 9edac369e0ae4c3c990bbb8dd845b27c P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:25.511996 2640 ts_tablet_manager.cc:1398] T 9edac369e0ae4c3c990bbb8dd845b27c P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:18:25.512683 30179 tablet_copy_service.cc:342] P 84cda20738d2452da166e31fc6da57a6: Request end of tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-9edac369e0ae4c3c990bbb8dd845b27c received from {username='slave'} at 127.20.207.2:56535
I20240502 08:18:25.512846 30179 tablet_copy_service.cc:433] P 84cda20738d2452da166e31fc6da57a6: ending tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-9edac369e0ae4c3c990bbb8dd845b27c on tablet 9edac369e0ae4c3c990bbb8dd845b27c with peer 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:25.584079 27516 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c2776f857a5f47e9b8022639dc54757d"
dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57898
I20240502 08:18:25.584293 27516 raft_consensus.cc:595] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:18:25.610661 2704 ts_tablet_manager.cc:902] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:25.622414 2704 tablet_copy_client.cc:308] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:25.623880 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet 5e0313e85d444e098bfa35be89a3ad97 from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 ({username='slave'} at 127.20.207.1:38081)
I20240502 08:18:25.623991 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet 5e0313e85d444e098bfa35be89a3ad97 from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 at {username='slave'} at 127.20.207.1:38081: session id = 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-5e0313e85d444e098bfa35be89a3ad97
I20240502 08:18:25.626922 27541 tablet_copy_source_session.cc:205] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:25.628302 2704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0313e85d444e098bfa35be89a3ad97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.630378 30423 consensus_queue.cc:237] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:25.632828 27652 raft_consensus.cc:1261] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:25.633330 2566 consensus_queue.cc:1035] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:25.634271 2704 tablet_copy_client.cc:783] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:25.634542 2640 ts_tablet_manager.cc:902] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1: Initiating tablet copy from peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:25.640554 2704 tablet_copy_client.cc:647] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:25.643142 30159 raft_consensus.cc:1261] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 08:18:25.644922 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c78e068ec3a451c892d58174b3f837c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:25.646185 2353 consensus_queue.cc:1035] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:25.647898 2053 raft_consensus.cc:1050] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442: attempting to promote NON_VOTER 84cda20738d2452da166e31fc6da57a6 to VOTER
I20240502 08:18:25.648552 2053 consensus_queue.cc:237] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:25.651125 2704 tablet_copy_client.cc:523] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:25.653739 2704 tablet_bootstrap.cc:492] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:25.658861 2566 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:25.663983 30156 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:25.672093 27115 catalog_manager.cc:5484] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 4 to 5, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:25.675725 2723 ts_tablet_manager.cc:902] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3: Initiating tablet copy from peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:25.693018 2720 ts_tablet_manager.cc:902] T c3a55662c6e346f0b73df334be4b14e3 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Initiating tablet copy from peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:25.700208 2723 tablet_copy_client.cc:308] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.7:40343
I20240502 08:18:25.700953 30310 tablet_copy_service.cc:140] P 49c5c8f70e494edd9fcda40cc0af6cc4: Received BeginTabletCopySession request for tablet f1f5650965e841ad98d857ca32299037 from peer 1f0bf4bc3c5146539146b825b048bff3 ({username='slave'} at 127.20.207.5:55543)
I20240502 08:18:25.701195 30310 tablet_copy_service.cc:161] P 49c5c8f70e494edd9fcda40cc0af6cc4: Beginning new tablet copy session on tablet f1f5650965e841ad98d857ca32299037 from peer 1f0bf4bc3c5146539146b825b048bff3 at {username='slave'} at 127.20.207.5:55543: session id = 1f0bf4bc3c5146539146b825b048bff3-f1f5650965e841ad98d857ca32299037
I20240502 08:18:25.710361 30310 tablet_copy_source_session.cc:205] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:25.711778 2723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f5650965e841ad98d857ca32299037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.712246 30156 raft_consensus.cc:1261] T a31709aa6b2b4ff784114fb48cbc8f2a P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:25.713475 30423 raft_consensus.cc:1261] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:25.715766 2730 ts_tablet_manager.cc:902] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: Initiating tablet copy from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:25.715124 2401 consensus_queue.cc:1035] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240502 08:18:25.716609 2719 ts_tablet_manager.cc:902] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:25.717545 2401 consensus_queue.cc:1035] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:25.719982 2723 tablet_copy_client.cc:783] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:25.720317 2723 tablet_copy_client.cc:647] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:25.721843 2723 tablet_copy_client.cc:523] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:25.727705 2723 tablet_bootstrap.cc:492] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap starting.
I20240502 08:18:25.747120 27516 raft_consensus.cc:1261] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:25.759232 2401 consensus_queue.cc:1035] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:18:25.759869 27584 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.8:41619, user_credentials={real_user=slave}} blocked reactor thread for 45484.1us
I20240502 08:18:25.777002 2723 tablet_bootstrap.cc:492] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:25.776585 27652 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:25.777434 2723 tablet_bootstrap.cc:492] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3: Bootstrap complete.
I20240502 08:18:25.780462 2723 ts_tablet_manager.cc:1367] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3: Time spent bootstrapping tablet: real 0.055s user 0.006s sys 0.003s
I20240502 08:18:25.781570 2723 raft_consensus.cc:348] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:25.787169 2723 raft_consensus.cc:729] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Initialized, Role: LEARNER
I20240502 08:18:25.787515 2640 tablet_copy_client.cc:308] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.5:37815
I20240502 08:18:25.787662 2723 consensus_queue.cc:260] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: NON_VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: true } }
I20240502 08:18:25.788082 30048 tablet_copy_service.cc:140] P 1f0bf4bc3c5146539146b825b048bff3: Received BeginTabletCopySession request for tablet 2e9b191ce9234c028b990efd74a59e2e from peer 83e171a5e16c4008923f1d43fc5ce7b1 ({username='slave'} at 127.20.207.2:49697)
I20240502 08:18:25.788223 30048 tablet_copy_service.cc:161] P 1f0bf4bc3c5146539146b825b048bff3: Beginning new tablet copy session on tablet 2e9b191ce9234c028b990efd74a59e2e from peer 83e171a5e16c4008923f1d43fc5ce7b1 at {username='slave'} at 127.20.207.2:49697: session id = 83e171a5e16c4008923f1d43fc5ce7b1-2e9b191ce9234c028b990efd74a59e2e
I20240502 08:18:25.790925 30048 tablet_copy_source_session.cc:205] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:25.792068 2640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e9b191ce9234c028b990efd74a59e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.796082 2540 raft_consensus.cc:979] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59: : Instructing follower 49c5c8f70e494edd9fcda40cc0af6cc4 to start an election
I20240502 08:18:25.796236 2540 raft_consensus.cc:1067] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Signalling peer 49c5c8f70e494edd9fcda40cc0af6cc4 to start an election
I20240502 08:18:25.796872 30289 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c2776f857a5f47e9b8022639dc54757d"
dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
from {username='slave'} at 127.20.207.3:49869
I20240502 08:18:25.797055 30289 raft_consensus.cc:482] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:25.797197 30289 raft_consensus.cc:3037] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:25.797417 2723 ts_tablet_manager.cc:1398] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240502 08:18:25.800479 30289 raft_consensus.cc:504] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:25.801980 30419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2776f857a5f47e9b8022639dc54757d" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:25.802203 30419 raft_consensus.cc:3037] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:25.803517 27516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2776f857a5f47e9b8022639dc54757d" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b01e7c8a302d41078fa39cfef665ff59"
I20240502 08:18:25.803678 27516 raft_consensus.cc:3032] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:25.803727 27516 raft_consensus.cc:729] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Running, Role: LEADER
I20240502 08:18:25.804003 27516 consensus_queue.cc:260] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:25.804387 27516 raft_consensus.cc:3037] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:25.805398 30419 raft_consensus.cc:2454] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 2.
I20240502 08:18:25.807643 30289 leader_election.cc:288] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 2 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:25.808092 27516 raft_consensus.cc:2454] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 2.
I20240502 08:18:25.808528 30226 leader_election.cc:302] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59; no voters:
I20240502 08:18:25.808636 30310 tablet_copy_service.cc:342] P 49c5c8f70e494edd9fcda40cc0af6cc4: Request end of tablet copy session 1f0bf4bc3c5146539146b825b048bff3-f1f5650965e841ad98d857ca32299037 received from {username='slave'} at 127.20.207.5:55543
I20240502 08:18:25.808769 30310 tablet_copy_service.cc:433] P 49c5c8f70e494edd9fcda40cc0af6cc4: ending tablet copy session 1f0bf4bc3c5146539146b825b048bff3-f1f5650965e841ad98d857ca32299037 on tablet f1f5650965e841ad98d857ca32299037 with peer 1f0bf4bc3c5146539146b825b048bff3
I20240502 08:18:25.808845 2588 raft_consensus.cc:2781] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:25.808995 2588 raft_consensus.cc:686] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 LEADER]: Becoming Leader. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:18:25.809309 2588 consensus_queue.cc:237] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:25.817620 2640 tablet_copy_client.cc:783] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:25.817831 2640 tablet_copy_client.cc:647] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:25.819957 2640 tablet_copy_client.cc:523] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:25.830225 27115 catalog_manager.cc:5484] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: term changed from 1 to 2, leader changed from b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) to 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7). New cstate: current_term: 2 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:25.831866 2640 tablet_bootstrap.cc:492] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
W20240502 08:18:25.834100 27587 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.3:44469, user_credentials={real_user=slave}} blocked reactor thread for 86372.1us
I20240502 08:18:25.838021 2401 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:25.846451 27115 catalog_manager.cc:5484] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:25.846489 30423 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:25.851567 27649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2776f857a5f47e9b8022639dc54757d" candidate_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:25.851758 27649 raft_consensus.cc:3037] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:25.858285 2730 tablet_copy_client.cc:308] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.8:41619
I20240502 08:18:25.858948 30443 tablet_copy_service.cc:140] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Received BeginTabletCopySession request for tablet d3a3473a93f14baaac48b9b7f47ac2a1 from peer 49c5c8f70e494edd9fcda40cc0af6cc4 ({username='slave'} at 127.20.207.7:47177)
I20240502 08:18:25.859107 30443 tablet_copy_service.cc:161] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Beginning new tablet copy session on tablet d3a3473a93f14baaac48b9b7f47ac2a1 from peer 49c5c8f70e494edd9fcda40cc0af6cc4 at {username='slave'} at 127.20.207.7:47177: session id = 49c5c8f70e494edd9fcda40cc0af6cc4-d3a3473a93f14baaac48b9b7f47ac2a1
I20240502 08:18:25.862597 27650 consensus_queue.cc:237] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:25.858798 27649 raft_consensus.cc:2454] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49c5c8f70e494edd9fcda40cc0af6cc4 in term 2.
I20240502 08:18:25.861987 30443 tablet_copy_source_session.cc:205] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:25.869679 2730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a3473a93f14baaac48b9b7f47ac2a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.874400 30156 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:25.877615 2730 tablet_copy_client.cc:783] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:25.878079 2730 tablet_copy_client.cc:647] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:25.889588 30156 raft_consensus.cc:1261] T a31709aa6b2b4ff784114fb48cbc8f2a P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:25.889915 2730 tablet_copy_client.cc:523] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:25.892416 2750 ts_tablet_manager.cc:902] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442: Initiating tablet copy from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:25.893338 2730 tablet_bootstrap.cc:492] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:25.901175 2735 consensus_queue.cc:1035] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:25.904073 2759 ts_tablet_manager.cc:902] T ca8d1416a54344feae081ed7b19dd1de P b01e7c8a302d41078fa39cfef665ff59: Initiating tablet copy from peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:25.905007 2720 tablet_copy_client.cc:308] T c3a55662c6e346f0b73df334be4b14e3 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.5:37815
I20240502 08:18:25.905974 30048 tablet_copy_service.cc:140] P 1f0bf4bc3c5146539146b825b048bff3: Received BeginTabletCopySession request for tablet c3a55662c6e346f0b73df334be4b14e3 from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 ({username='slave'} at 127.20.207.1:49229)
I20240502 08:18:25.906442 30048 tablet_copy_service.cc:161] P 1f0bf4bc3c5146539146b825b048bff3: Beginning new tablet copy session on tablet c3a55662c6e346f0b73df334be4b14e3 from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 at {username='slave'} at 127.20.207.1:49229: session id = 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-c3a55662c6e346f0b73df334be4b14e3
I20240502 08:18:25.909792 30048 tablet_copy_source_session.cc:205] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:25.912087 2719 tablet_copy_client.cc:308] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:25.913967 30419 raft_consensus.cc:1261] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:25.914803 2735 consensus_queue.cc:1035] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:25.919972 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet de4b822c4aca4f7f92b73cc9279ddec5 from peer bf8ac71b15f740a0a928911360773442 ({username='slave'} at 127.20.207.4:32927)
I20240502 08:18:25.920122 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet de4b822c4aca4f7f92b73cc9279ddec5 from peer bf8ac71b15f740a0a928911360773442 at {username='slave'} at 127.20.207.4:32927: session id = bf8ac71b15f740a0a928911360773442-de4b822c4aca4f7f92b73cc9279ddec5
I20240502 08:18:25.921308 2720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a55662c6e346f0b73df334be4b14e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.922777 27541 tablet_copy_source_session.cc:205] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:25.925797 2735 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:25.930938 2720 tablet_copy_client.cc:783] T c3a55662c6e346f0b73df334be4b14e3 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:25.931154 2720 tablet_copy_client.cc:647] T c3a55662c6e346f0b73df334be4b14e3 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:25.934793 30156 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:25.940466 27115 catalog_manager.cc:5484] T a31709aa6b2b4ff784114fb48cbc8f2a P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 8 to 9, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 1 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:25.948508 2719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de4b822c4aca4f7f92b73cc9279ddec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.949245 27498 tablet_service.cc:1514] Processing DeleteTablet for tablet a31709aa6b2b4ff784114fb48cbc8f2a with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:25.951928 2640 tablet_bootstrap.cc:492] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:25.952276 2640 tablet_bootstrap.cc:492] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap complete.
I20240502 08:18:25.952497 2640 ts_tablet_manager.cc:1367] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.121s user 0.009s sys 0.001s
I20240502 08:18:25.954252 2730 tablet_bootstrap.cc:492] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:25.954811 2730 tablet_bootstrap.cc:492] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap complete.
I20240502 08:18:25.955055 2730 ts_tablet_manager.cc:1367] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.062s user 0.009s sys 0.003s
I20240502 08:18:25.955754 2730 raft_consensus.cc:348] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:25.955984 2730 raft_consensus.cc:729] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: LEARNER
I20240502 08:18:25.956126 2719 tablet_copy_client.cc:783] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:25.956193 2730 consensus_queue.cc:260] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:25.953145 2640 raft_consensus.cc:348] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:25.956897 2640 raft_consensus.cc:729] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: LEARNER
I20240502 08:18:25.956367 2719 tablet_copy_client.cc:647] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:25.957078 2640 consensus_queue.cc:260] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:25.957901 2767 tablet_replica.cc:330] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:25.958141 2767 raft_consensus.cc:2229] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:25.958367 2767 pending_rounds.cc:62] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59: Trying to abort 1 pending ops.
I20240502 08:18:25.958495 2719 tablet_copy_client.cc:523] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:25.958674 2767 pending_rounds.cc:68] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023156861534765056 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "a31709aa6b2b4ff784114fb48cbc8f2a" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } } }
I20240502 08:18:25.959622 2767 raft_consensus.cc:2894] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240502 08:18:25.959892 2767 raft_consensus.cc:2866] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 08:18:25.960124 2767 raft_consensus.cc:2258] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:25.961194 2704 tablet_bootstrap.cc:492] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:25.962451 2719 tablet_bootstrap.cc:492] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:25.964959 2704 tablet_bootstrap.cc:492] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap complete.
I20240502 08:18:25.965219 2704 ts_tablet_manager.cc:1367] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.312s user 0.064s sys 0.028s
I20240502 08:18:25.965837 2704 raft_consensus.cc:348] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:25.966104 2704 raft_consensus.cc:729] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: LEARNER
I20240502 08:18:25.966307 2704 consensus_queue.cc:260] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:25.971087 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a31709aa6b2b4ff784114fb48cbc8f2a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:25.971938 2640 ts_tablet_manager.cc:1398] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.019s user 0.002s sys 0.002s
I20240502 08:18:25.972694 30048 tablet_copy_service.cc:342] P 1f0bf4bc3c5146539146b825b048bff3: Request end of tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-2e9b191ce9234c028b990efd74a59e2e received from {username='slave'} at 127.20.207.2:49697
I20240502 08:18:25.972855 30048 tablet_copy_service.cc:433] P 1f0bf4bc3c5146539146b825b048bff3: ending tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-2e9b191ce9234c028b990efd74a59e2e on tablet 2e9b191ce9234c028b990efd74a59e2e with peer 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:25.975029 2704 ts_tablet_manager.cc:1398] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 08:18:25.981904 2730 ts_tablet_manager.cc:1398] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240502 08:18:25.984580 30443 tablet_copy_service.cc:342] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Request end of tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-d3a3473a93f14baaac48b9b7f47ac2a1 received from {username='slave'} at 127.20.207.7:47177
I20240502 08:18:25.984700 30443 tablet_copy_service.cc:433] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: ending tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-d3a3473a93f14baaac48b9b7f47ac2a1 on tablet d3a3473a93f14baaac48b9b7f47ac2a1 with peer 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:25.985787 30419 raft_consensus.cc:2932] T a31709aa6b2b4ff784114fb48cbc8f2a P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:25.976801 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-5e0313e85d444e098bfa35be89a3ad97 received from {username='slave'} at 127.20.207.1:38081
I20240502 08:18:26.000718 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-5e0313e85d444e098bfa35be89a3ad97 on tablet 5e0313e85d444e098bfa35be89a3ad97 with peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60
I20240502 08:18:26.001976 2767 ts_tablet_manager.cc:1875] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:26.005676 2719 tablet_bootstrap.cc:492] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:26.006075 2719 tablet_bootstrap.cc:492] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442: Bootstrap complete.
I20240502 08:18:26.006285 2719 ts_tablet_manager.cc:1367] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.005s
I20240502 08:18:26.006878 2719 raft_consensus.cc:348] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:26.007122 2719 raft_consensus.cc:729] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: LEARNER
I20240502 08:18:26.007302 2719 consensus_queue.cc:260] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:26.009930 2720 tablet_copy_client.cc:523] T c3a55662c6e346f0b73df334be4b14e3 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:26.013403 2720 tablet_bootstrap.cc:492] T c3a55662c6e346f0b73df334be4b14e3 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:26.014710 2759 tablet_copy_client.cc:308] T ca8d1416a54344feae081ed7b19dd1de P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.7:40343
I20240502 08:18:26.016299 30310 tablet_copy_service.cc:140] P 49c5c8f70e494edd9fcda40cc0af6cc4: Received BeginTabletCopySession request for tablet ca8d1416a54344feae081ed7b19dd1de from peer b01e7c8a302d41078fa39cfef665ff59 ({username='slave'} at 127.20.207.3:49869)
I20240502 08:18:26.016446 30310 tablet_copy_service.cc:161] P 49c5c8f70e494edd9fcda40cc0af6cc4: Beginning new tablet copy session on tablet ca8d1416a54344feae081ed7b19dd1de from peer b01e7c8a302d41078fa39cfef665ff59 at {username='slave'} at 127.20.207.3:49869: session id = b01e7c8a302d41078fa39cfef665ff59-ca8d1416a54344feae081ed7b19dd1de
I20240502 08:18:26.016691 2719 ts_tablet_manager.cc:1398] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240502 08:18:26.017222 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session bf8ac71b15f740a0a928911360773442-de4b822c4aca4f7f92b73cc9279ddec5 received from {username='slave'} at 127.20.207.4:32927
I20240502 08:18:26.017306 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session bf8ac71b15f740a0a928911360773442-de4b822c4aca4f7f92b73cc9279ddec5 on tablet de4b822c4aca4f7f92b73cc9279ddec5 with peer bf8ac71b15f740a0a928911360773442
I20240502 08:18:26.019194 30310 tablet_copy_source_session.cc:205] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:26.020339 2759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8d1416a54344feae081ed7b19dd1de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.020385 27388 consensus_queue.cc:237] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:26.025790 2759 tablet_copy_client.cc:783] T ca8d1416a54344feae081ed7b19dd1de P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:26.025976 2759 tablet_copy_client.cc:647] T ca8d1416a54344feae081ed7b19dd1de P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:26.028470 2720 tablet_bootstrap.cc:492] T c3a55662c6e346f0b73df334be4b14e3 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:26.029012 2720 tablet_bootstrap.cc:492] T c3a55662c6e346f0b73df334be4b14e3 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap complete.
I20240502 08:18:26.029209 2720 ts_tablet_manager.cc:1367] T c3a55662c6e346f0b73df334be4b14e3 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.002s
I20240502 08:18:26.029687 2720 raft_consensus.cc:348] T c3a55662c6e346f0b73df334be4b14e3 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:26.029986 2720 raft_consensus.cc:729] T c3a55662c6e346f0b73df334be4b14e3 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: LEARNER
I20240502 08:18:26.030167 2720 consensus_queue.cc:260] T c3a55662c6e346f0b73df334be4b14e3 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:26.030619 2720 ts_tablet_manager.cc:1398] T c3a55662c6e346f0b73df334be4b14e3 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 08:18:26.033102 30048 tablet_copy_service.cc:342] P 1f0bf4bc3c5146539146b825b048bff3: Request end of tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-c3a55662c6e346f0b73df334be4b14e3 received from {username='slave'} at 127.20.207.1:49229
I20240502 08:18:26.033249 30048 tablet_copy_service.cc:433] P 1f0bf4bc3c5146539146b825b048bff3: ending tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-c3a55662c6e346f0b73df334be4b14e3 on tablet c3a55662c6e346f0b73df334be4b14e3 with peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60
I20240502 08:18:26.039891 2759 tablet_copy_client.cc:523] T ca8d1416a54344feae081ed7b19dd1de P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:26.041496 27516 raft_consensus.cc:1261] T 0ed6b8adc62f471696cbf0330524e353 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:26.041993 2513 consensus_queue.cc:1035] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:26.043691 2759 tablet_bootstrap.cc:492] T ca8d1416a54344feae081ed7b19dd1de P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:26.043779 27257 raft_consensus.cc:1203] T c3a55662c6e346f0b73df334be4b14e3 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:26.048187 2513 raft_consensus.cc:2932] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:26.051967 2759 tablet_bootstrap.cc:492] T ca8d1416a54344feae081ed7b19dd1de P b01e7c8a302d41078fa39cfef665ff59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:26.052232 2759 tablet_bootstrap.cc:492] T ca8d1416a54344feae081ed7b19dd1de P b01e7c8a302d41078fa39cfef665ff59: Bootstrap complete.
I20240502 08:18:26.052425 2759 ts_tablet_manager.cc:1367] T ca8d1416a54344feae081ed7b19dd1de P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240502 08:18:26.053181 2759 raft_consensus.cc:348] T ca8d1416a54344feae081ed7b19dd1de P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:26.053382 2759 raft_consensus.cc:729] T ca8d1416a54344feae081ed7b19dd1de P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: LEARNER
I20240502 08:18:26.053543 2759 consensus_queue.cc:260] T ca8d1416a54344feae081ed7b19dd1de P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:26.053792 27518 raft_consensus.cc:2932] T 0ed6b8adc62f471696cbf0330524e353 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
W20240502 08:18:26.055497 27325 consensus_peers.cc:485] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): Couldn't send request to peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ed6b8adc62f471696cbf0330524e353. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:26.058254 2759 ts_tablet_manager.cc:1398] T ca8d1416a54344feae081ed7b19dd1de P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:18:26.059139 30310 tablet_copy_service.cc:342] P 49c5c8f70e494edd9fcda40cc0af6cc4: Request end of tablet copy session b01e7c8a302d41078fa39cfef665ff59-ca8d1416a54344feae081ed7b19dd1de received from {username='slave'} at 127.20.207.3:49869
I20240502 08:18:26.059692 30310 tablet_copy_service.cc:433] P 49c5c8f70e494edd9fcda40cc0af6cc4: ending tablet copy session b01e7c8a302d41078fa39cfef665ff59-ca8d1416a54344feae081ed7b19dd1de on tablet ca8d1416a54344feae081ed7b19dd1de with peer b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:26.060956 30156 raft_consensus.cc:1261] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:26.061641 2758 consensus_queue.cc:1035] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:26.064240 30156 raft_consensus.cc:2932] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:26.070142 27115 catalog_manager.cc:5484] T 0ed6b8adc62f471696cbf0330524e353 P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index -1 to 2, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:26.081792 2750 tablet_copy_client.cc:308] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.8:41619
I20240502 08:18:26.081946 27388 raft_consensus.cc:1203] T 9edac369e0ae4c3c990bbb8dd845b27c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:26.082870 30443 tablet_copy_service.cc:140] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Received BeginTabletCopySession request for tablet 73548c1dd45f4c31bf3caf89b6d9ebac from peer bf8ac71b15f740a0a928911360773442 ({username='slave'} at 127.20.207.4:55985)
I20240502 08:18:26.083015 30443 tablet_copy_service.cc:161] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Beginning new tablet copy session on tablet 73548c1dd45f4c31bf3caf89b6d9ebac from peer bf8ac71b15f740a0a928911360773442 at {username='slave'} at 127.20.207.4:55985: session id = bf8ac71b15f740a0a928911360773442-73548c1dd45f4c31bf3caf89b6d9ebac
I20240502 08:18:26.085870 30443 tablet_copy_source_session.cc:205] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:26.089247 2767 ts_tablet_manager.cc:1888] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:18:26.089358 2767 log.cc:1196] T a31709aa6b2b4ff784114fb48cbc8f2a P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/a31709aa6b2b4ff784114fb48cbc8f2a
I20240502 08:18:26.091482 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet a31709aa6b2b4ff784114fb48cbc8f2a (table rebalance_test_table_17 [id=c179f8e244074d9ead49cbf91748c01e]) successfully deleted
I20240502 08:18:26.093067 2750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73548c1dd45f4c31bf3caf89b6d9ebac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.099937 30156 consensus_queue.cc:237] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:26.102768 27652 raft_consensus.cc:1261] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:26.103030 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51201cc579394a2b819278a5943485a0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:26.103533 30419 raft_consensus.cc:1261] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:26.103829 2688 consensus_queue.cc:1035] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:26.104350 2750 tablet_copy_client.cc:783] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:26.104586 2750 tablet_copy_client.cc:647] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:26.105592 2688 consensus_queue.cc:1035] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:26.106571 2750 tablet_copy_client.cc:523] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:26.111497 2750 tablet_bootstrap.cc:492] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442: Bootstrap starting.
I20240502 08:18:26.118629 2457 raft_consensus.cc:2932] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:26.124644 30423 raft_consensus.cc:2932] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:26.127030 27115 catalog_manager.cc:5484] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:26.133968 2759 ts_tablet_manager.cc:902] T 4c78e068ec3a451c892d58174b3f837c P b01e7c8a302d41078fa39cfef665ff59: Initiating tablet copy from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:26.143391 30028 raft_consensus.cc:1203] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:26.143352 27652 raft_consensus.cc:2932] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:26.154233 2750 tablet_bootstrap.cc:492] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:26.154598 2750 tablet_bootstrap.cc:492] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442: Bootstrap complete.
I20240502 08:18:26.154825 2750 ts_tablet_manager.cc:1367] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.003s
I20240502 08:18:26.156025 2750 raft_consensus.cc:348] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:26.156227 2750 raft_consensus.cc:729] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf8ac71b15f740a0a928911360773442, State: Initialized, Role: LEARNER
I20240502 08:18:26.156394 2750 consensus_queue.cc:260] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: NON_VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: true } }
I20240502 08:18:26.157056 2750 ts_tablet_manager.cc:1398] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:26.157814 30443 tablet_copy_service.cc:342] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Request end of tablet copy session bf8ac71b15f740a0a928911360773442-73548c1dd45f4c31bf3caf89b6d9ebac received from {username='slave'} at 127.20.207.4:55985
I20240502 08:18:26.157949 30443 tablet_copy_service.cc:433] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: ending tablet copy session bf8ac71b15f740a0a928911360773442-73548c1dd45f4c31bf3caf89b6d9ebac on tablet 73548c1dd45f4c31bf3caf89b6d9ebac with peer bf8ac71b15f740a0a928911360773442
I20240502 08:18:26.161414 27649 raft_consensus.cc:1203] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:18:26.177870 2759 tablet_copy_client.cc:308] T 4c78e068ec3a451c892d58174b3f837c P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.8:41619
I20240502 08:18:26.179585 30443 tablet_copy_service.cc:140] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Received BeginTabletCopySession request for tablet 4c78e068ec3a451c892d58174b3f837c from peer b01e7c8a302d41078fa39cfef665ff59 ({username='slave'} at 127.20.207.3:40171)
I20240502 08:18:26.179749 30443 tablet_copy_service.cc:161] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Beginning new tablet copy session on tablet 4c78e068ec3a451c892d58174b3f837c from peer b01e7c8a302d41078fa39cfef665ff59 at {username='slave'} at 127.20.207.3:40171: session id = b01e7c8a302d41078fa39cfef665ff59-4c78e068ec3a451c892d58174b3f837c
I20240502 08:18:26.181654 30443 tablet_copy_source_session.cc:205] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:26.182849 2759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c78e068ec3a451c892d58174b3f837c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.187553 27388 raft_consensus.cc:1203] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:26.188210 2759 tablet_copy_client.cc:783] T 4c78e068ec3a451c892d58174b3f837c P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:26.188417 2759 tablet_copy_client.cc:647] T 4c78e068ec3a451c892d58174b3f837c P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:26.191310 27516 consensus_queue.cc:237] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:26.192018 2759 tablet_copy_client.cc:523] T 4c78e068ec3a451c892d58174b3f837c P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:26.193622 30028 raft_consensus.cc:1261] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:26.193621 30423 raft_consensus.cc:1261] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:26.194105 27456 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 -> Peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Couldn't send request to peer 84cda20738d2452da166e31fc6da57a6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d122f68d36024614a017e8cc3a1224c1. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:26.194254 2529 consensus_queue.cc:1035] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:26.194892 2529 consensus_queue.cc:1035] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:26.199252 2759 tablet_bootstrap.cc:492] T 4c78e068ec3a451c892d58174b3f837c P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:26.199926 27255 raft_consensus.cc:1203] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:18:26.207053 2529 raft_consensus.cc:2932] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:26.213886 27649 raft_consensus.cc:1261] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:26.215688 2588 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:26.224387 30423 raft_consensus.cc:2932] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:26.229933 30028 raft_consensus.cc:2932] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:26.231163 2759 tablet_bootstrap.cc:492] T 4c78e068ec3a451c892d58174b3f837c P b01e7c8a302d41078fa39cfef665ff59: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:26.231477 2759 tablet_bootstrap.cc:492] T 4c78e068ec3a451c892d58174b3f837c P b01e7c8a302d41078fa39cfef665ff59: Bootstrap complete.
I20240502 08:18:26.231678 2759 ts_tablet_manager.cc:1367] T 4c78e068ec3a451c892d58174b3f837c P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.000s
I20240502 08:18:26.232271 2759 raft_consensus.cc:348] T 4c78e068ec3a451c892d58174b3f837c P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:26.232484 2759 raft_consensus.cc:729] T 4c78e068ec3a451c892d58174b3f837c P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: LEARNER
I20240502 08:18:26.232631 2759 consensus_queue.cc:260] T 4c78e068ec3a451c892d58174b3f837c P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:26.236279 2759 ts_tablet_manager.cc:1398] T 4c78e068ec3a451c892d58174b3f837c P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:18:26.237367 30443 tablet_copy_service.cc:342] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Request end of tablet copy session b01e7c8a302d41078fa39cfef665ff59-4c78e068ec3a451c892d58174b3f837c received from {username='slave'} at 127.20.207.3:40171
I20240502 08:18:26.237509 30443 tablet_copy_service.cc:433] P 31a1e33dcdbf42f58ea72ce1c0afb0ad: ending tablet copy session b01e7c8a302d41078fa39cfef665ff59-4c78e068ec3a451c892d58174b3f837c on tablet 4c78e068ec3a451c892d58174b3f837c with peer b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:26.238333 27115 catalog_manager.cc:5484] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index -1 to 2, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:26.251396 27652 raft_consensus.cc:1203] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:26.258487 27516 consensus_queue.cc:237] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:26.263269 30028 raft_consensus.cc:1261] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:26.263387 27453 consensus_peers.cc:485] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e80b9a98bdd041b1ac97af055951a92a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:26.263420 27255 raft_consensus.cc:1261] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:26.263926 2800 consensus_queue.cc:1035] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:26.264411 2800 consensus_queue.cc:1035] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:26.266880 2540 raft_consensus.cc:2932] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:26.270952 30028 raft_consensus.cc:2932] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:26.275959 30288 raft_consensus.cc:1203] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:26.280423 27108 catalog_manager.cc:5484] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 reported cstate change: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:26.284299 27516 raft_consensus.cc:1261] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 2 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:26.284693 27518 raft_consensus.cc:1203] T ca8d1416a54344feae081ed7b19dd1de P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:26.295437 2588 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:26.295725 27255 raft_consensus.cc:2932] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:26.321439 30423 raft_consensus.cc:1261] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:26.322064 2588 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:26.363185 30156 consensus_queue.cc:237] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:26.365293 27388 raft_consensus.cc:1261] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:26.365306 27518 raft_consensus.cc:1261] T 4165b86bc05e47f6a9803fb45ca01569 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:26.365522 30093 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): Couldn't send request to peer 31a1e33dcdbf42f58ea72ce1c0afb0ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4165b86bc05e47f6a9803fb45ca01569. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:26.365864 2457 consensus_queue.cc:1035] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:26.366182 2688 consensus_queue.cc:1035] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:26.369982 2457 raft_consensus.cc:2932] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:26.372664 27388 raft_consensus.cc:2932] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:26.373908 27518 raft_consensus.cc:2932] T 4165b86bc05e47f6a9803fb45ca01569 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:26.379679 27115 catalog_manager.cc:5484] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index -1 to 2, NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) added. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } }
I20240502 08:18:26.563477 2910 ts_tablet_manager.cc:902] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1: Initiating tablet copy from peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:26.564818 2540 raft_consensus.cc:1050] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER bf8ac71b15f740a0a928911360773442 to VOTER
I20240502 08:18:26.565623 2540 consensus_queue.cc:237] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:26.566691 2910 tablet_copy_client.cc:308] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.6:38899
I20240502 08:18:26.567389 30179 tablet_copy_service.cc:140] P 84cda20738d2452da166e31fc6da57a6: Received BeginTabletCopySession request for tablet 51201cc579394a2b819278a5943485a0 from peer 83e171a5e16c4008923f1d43fc5ce7b1 ({username='slave'} at 127.20.207.2:56535)
I20240502 08:18:26.567615 30179 tablet_copy_service.cc:161] P 84cda20738d2452da166e31fc6da57a6: Beginning new tablet copy session on tablet 51201cc579394a2b819278a5943485a0 from peer 83e171a5e16c4008923f1d43fc5ce7b1 at {username='slave'} at 127.20.207.2:56535: session id = 83e171a5e16c4008923f1d43fc5ce7b1-51201cc579394a2b819278a5943485a0
I20240502 08:18:26.571130 30289 raft_consensus.cc:1261] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:26.571373 27255 raft_consensus.cc:1261] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:26.571913 2800 consensus_queue.cc:1035] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:26.572252 30179 tablet_copy_source_session.cc:205] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:26.572268 2813 consensus_queue.cc:1035] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:26.572459 27652 raft_consensus.cc:1261] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:26.581107 2589 raft_consensus.cc:1050] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4: attempting to promote NON_VOTER 1f0bf4bc3c5146539146b825b048bff3 to VOTER
I20240502 08:18:26.581961 2589 consensus_queue.cc:237] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:26.582190 2540 consensus_queue.cc:1035] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:26.588382 2814 raft_consensus.cc:1050] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 to VOTER
I20240502 08:18:26.589350 2814 consensus_queue.cc:237] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:26.593287 2910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51201cc579394a2b819278a5943485a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.594213 30028 raft_consensus.cc:1261] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEARNER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:26.594877 2588 consensus_queue.cc:1035] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:26.597338 30025 raft_consensus.cc:1261] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:26.597627 27388 raft_consensus.cc:1261] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:26.598083 2814 consensus_queue.cc:1035] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:26.598598 27257 raft_consensus.cc:1261] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:26.599963 2928 ts_tablet_manager.cc:902] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:26.600032 2540 consensus_queue.cc:1035] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:26.600540 2540 consensus_queue.cc:1035] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:26.602288 30159 raft_consensus.cc:1261] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:26.602929 2588 consensus_queue.cc:1035] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:26.613494 30423 raft_consensus.cc:1261] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:26.614209 2588 consensus_queue.cc:1035] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:26.621654 2813 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:26.624594 30025 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:26.641721 2281 raft_consensus.cc:1050] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3: attempting to promote NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 to VOTER
I20240502 08:18:26.643167 2281 consensus_queue.cc:237] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:26.647291 2520 raft_consensus.cc:1050] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6: attempting to promote NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 to VOTER
I20240502 08:18:26.648381 2520 consensus_queue.cc:237] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:26.653775 2937 ts_tablet_manager.cc:902] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Initiating tablet copy from peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:26.660238 30288 raft_consensus.cc:1261] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:26.660909 2588 raft_consensus.cc:2932] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:26.661806 2457 consensus_queue.cc:1035] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:26.662770 27389 raft_consensus.cc:1261] T 9edac369e0ae4c3c990bbb8dd845b27c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:26.663479 2688 consensus_queue.cc:1035] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:26.671989 27258 raft_consensus.cc:1261] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:26.672675 2457 consensus_queue.cc:1035] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:26.677495 27114 catalog_manager.cc:5484] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 2 to 3, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:26.690258 30159 raft_consensus.cc:2932] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:26.701871 27257 raft_consensus.cc:1261] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:26.703128 2550 consensus_queue.cc:1035] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:26.705963 2457 raft_consensus.cc:2932] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:26.706106 27256 raft_consensus.cc:2932] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:26.708595 30025 raft_consensus.cc:2932] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:26.714071 27388 raft_consensus.cc:1261] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:26.717918 27114 catalog_manager.cc:5484] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 7 to 8, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:26.719516 2281 raft_consensus.cc:1050] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3: attempting to promote NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 to VOTER
I20240502 08:18:26.720211 2281 consensus_queue.cc:237] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:26.722095 2800 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:26.718358 27112 catalog_manager.cc:5484] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:26.729928 27649 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:26.731407 27518 raft_consensus.cc:1203] T 4c78e068ec3a451c892d58174b3f837c P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:18:26.732949 30423 raft_consensus.cc:2932] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:26.740818 30290 raft_consensus.cc:2932] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:26.744205 27112 catalog_manager.cc:5484] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 5 to 6, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:26.745366 27388 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:26.746721 27258 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:26.768934 2453 raft_consensus.cc:1050] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad: attempting to promote NON_VOTER bf8ac71b15f740a0a928911360773442 to VOTER
I20240502 08:18:26.769769 2453 consensus_queue.cc:237] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:26.777719 2928 tablet_copy_client.cc:308] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:26.779881 2937 tablet_copy_client.cc:308] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.2:46291
I20240502 08:18:26.780517 27410 tablet_copy_service.cc:140] P 83e171a5e16c4008923f1d43fc5ce7b1: Received BeginTabletCopySession request for tablet 0ed6b8adc62f471696cbf0330524e353 from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 ({username='slave'} at 127.20.207.1:38029)
I20240502 08:18:26.780638 27410 tablet_copy_service.cc:161] P 83e171a5e16c4008923f1d43fc5ce7b1: Beginning new tablet copy session on tablet 0ed6b8adc62f471696cbf0330524e353 from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 at {username='slave'} at 127.20.207.1:38029: session id = 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-0ed6b8adc62f471696cbf0330524e353
I20240502 08:18:26.783234 27410 tablet_copy_source_session.cc:205] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:26.754312 30289 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:26.787266 2937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed6b8adc62f471696cbf0330524e353. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.790939 2944 consensus_queue.cc:1035] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:26.795821 27257 raft_consensus.cc:1261] T c3a55662c6e346f0b73df334be4b14e3 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:26.798658 27389 raft_consensus.cc:1261] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:26.798935 2937 tablet_copy_client.cc:783] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:26.799332 2937 tablet_copy_client.cc:647] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:26.800908 2937 tablet_copy_client.cc:523] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:26.806790 2937 tablet_bootstrap.cc:492] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
I20240502 08:18:26.808429 27389 raft_consensus.cc:2932] T 9edac369e0ae4c3c990bbb8dd845b27c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:26.822683 2353 raft_consensus.cc:1050] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: attempting to promote NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 to VOTER
I20240502 08:18:26.823573 2353 consensus_queue.cc:237] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:26.825850 2963 ts_tablet_manager.cc:902] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4: Initiating tablet copy from peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:26.831260 30287 raft_consensus.cc:1261] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:26.832605 2281 consensus_queue.cc:1035] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:26.844223 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet d122f68d36024614a017e8cc3a1224c1 from peer 84cda20738d2452da166e31fc6da57a6 ({username='slave'} at 127.20.207.6:40325)
I20240502 08:18:26.844385 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet d122f68d36024614a017e8cc3a1224c1 from peer 84cda20738d2452da166e31fc6da57a6 at {username='slave'} at 127.20.207.6:40325: session id = 84cda20738d2452da166e31fc6da57a6-d122f68d36024614a017e8cc3a1224c1
I20240502 08:18:26.848326 27541 tablet_copy_source_session.cc:205] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:26.855401 2281 consensus_queue.cc:1035] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:26.857018 2281 consensus_queue.cc:1035] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:26.860390 2928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d122f68d36024614a017e8cc3a1224c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.866814 2928 tablet_copy_client.cc:783] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:26.867270 2928 tablet_copy_client.cc:647] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:26.871220 2928 tablet_copy_client.cc:523] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:26.875339 2821 raft_consensus.cc:1050] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4: attempting to promote NON_VOTER b01e7c8a302d41078fa39cfef665ff59 to VOTER
I20240502 08:18:26.876026 2821 consensus_queue.cc:237] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:26.877686 27518 consensus_queue.cc:237] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:26.882329 27386 raft_consensus.cc:1261] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:26.882932 27258 raft_consensus.cc:1261] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:26.883018 2800 consensus_queue.cc:1035] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:26.884549 2540 consensus_queue.cc:1035] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:26.887383 2540 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:26.888703 27258 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:26.890946 27257 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:26.901602 27518 consensus_queue.cc:237] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:26.904291 27649 raft_consensus.cc:1261] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:26.905076 2529 consensus_queue.cc:1035] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:26.905354 2928 tablet_bootstrap.cc:492] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:26.908830 2529 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:26.911363 27650 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:26.911708 27115 catalog_manager.cc:5484] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 8 to 9, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:26.916080 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5e0313e85d444e098bfa35be89a3ad97 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:26.918098 30290 raft_consensus.cc:1261] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:26.919852 27108 catalog_manager.cc:5484] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 6 to 7, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:26.924813 27388 raft_consensus.cc:2932] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:26.926993 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet de4b822c4aca4f7f92b73cc9279ddec5 with delete_type TABLET_DATA_TOMBSTONED (TS 49c5c8f70e494edd9fcda40cc0af6cc4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:26.929219 2965 consensus_queue.cc:1035] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:26.934589 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet de4b822c4aca4f7f92b73cc9279ddec5 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:26.944989 30007 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0313e85d444e098bfa35be89a3ad97 with delete_type TABLET_DATA_TOMBSTONED (TS 1f0bf4bc3c5146539146b825b048bff3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:26.945523 2944 raft_consensus.cc:2932] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:26.953164 30288 consensus_queue.cc:237] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:26.955474 30423 raft_consensus.cc:1261] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:26.957041 2976 tablet_replica.cc:330] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:26.957069 2589 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240502 08:18:26.957653 2976 raft_consensus.cc:2229] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:26.958065 2937 tablet_bootstrap.cc:492] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:26.957965 2976 raft_consensus.cc:2258] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:26.959604 2589 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:26.960503 27386 raft_consensus.cc:1261] T d3a3473a93f14baaac48b9b7f47ac2a1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:26.961045 2968 consensus_queue.cc:1035] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:26.962596 30423 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:26.963394 27389 raft_consensus.cc:2932] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:26.966513 27255 raft_consensus.cc:2932] T c3a55662c6e346f0b73df334be4b14e3 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:26.967584 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c2776f857a5f47e9b8022639dc54757d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:26.971305 27108 catalog_manager.cc:5484] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 8 to 10, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 2 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:26.974956 27498 tablet_service.cc:1514] Processing DeleteTablet for tablet c2776f857a5f47e9b8022639dc54757d with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:26.971320 27115 catalog_manager.cc:5484] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:26.981817 27258 raft_consensus.cc:1261] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:26.982524 2813 consensus_queue.cc:1035] T de4b822c4aca4f7f92b73cc9279ddec5 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:26.984577 2976 ts_tablet_manager.cc:1875] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:26.984691 2937 tablet_bootstrap.cc:492] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap complete.
I20240502 08:18:26.984984 27518 raft_consensus.cc:1261] T ca8d1416a54344feae081ed7b19dd1de P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:26.985028 2937 ts_tablet_manager.cc:1367] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.178s user 0.011s sys 0.011s
I20240502 08:18:26.985548 2589 consensus_queue.cc:1035] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:26.989503 2980 tablet_replica.cc:330] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:26.992544 2980 raft_consensus.cc:2229] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:26.993038 2980 raft_consensus.cc:2258] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240502 08:18:26.993203 30358 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.3:44469, user_credentials={real_user=slave}} blocked reactor thread for 171996us
I20240502 08:18:26.994741 30290 raft_consensus.cc:1261] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:26.994889 2937 raft_consensus.cc:348] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:26.995149 2937 raft_consensus.cc:729] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: LEARNER
I20240502 08:18:26.995707 2566 consensus_queue.cc:1035] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:26.995589 2937 consensus_queue.cc:260] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:26.994060 27258 raft_consensus.cc:2932] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:26.997104 30423 raft_consensus.cc:1261] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:26.996902 2943 raft_consensus.cc:2932] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:26.997742 27649 raft_consensus.cc:1261] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442 [term 1 LEARNER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:26.997704 2589 consensus_queue.cc:1035] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:27.000396 2937 ts_tablet_manager.cc:1398] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240502 08:18:27.001066 27410 tablet_copy_service.cc:342] P 83e171a5e16c4008923f1d43fc5ce7b1: Request end of tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-0ed6b8adc62f471696cbf0330524e353 received from {username='slave'} at 127.20.207.1:38029
I20240502 08:18:27.001186 27410 tablet_copy_service.cc:433] P 83e171a5e16c4008923f1d43fc5ce7b1: ending tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-0ed6b8adc62f471696cbf0330524e353 on tablet 0ed6b8adc62f471696cbf0330524e353 with peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60
I20240502 08:18:27.002331 27387 raft_consensus.cc:2932] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:27.006064 27255 raft_consensus.cc:2932] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:27.007172 27110 catalog_manager.cc:5484] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:27.010229 2910 tablet_copy_client.cc:783] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:27.010529 2910 tablet_copy_client.cc:647] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:27.019274 27649 raft_consensus.cc:1261] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:27.021543 27518 raft_consensus.cc:1261] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:27.021663 2980 ts_tablet_manager.cc:1875] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:27.023670 27649 raft_consensus.cc:1261] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:27.024171 2589 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:27.024327 30159 consensus_queue.cc:237] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:27.025173 2589 consensus_queue.cc:1035] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:27.026561 27387 raft_consensus.cc:1261] T 9edac369e0ae4c3c990bbb8dd845b27c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:27.027323 27257 raft_consensus.cc:1261] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:27.027966 2520 consensus_queue.cc:1035] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:27.028576 2520 consensus_queue.cc:1035] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:27.029141 2910 tablet_copy_client.cc:523] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:27.031733 2520 raft_consensus.cc:2932] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:27.034883 27387 raft_consensus.cc:2932] T 9edac369e0ae4c3c990bbb8dd845b27c P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:27.036223 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9edac369e0ae4c3c990bbb8dd845b27c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:27.037899 2910 tablet_bootstrap.cc:492] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap starting.
I20240502 08:18:27.039383 30419 raft_consensus.cc:1261] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:27.040462 30159 raft_consensus.cc:1261] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:27.041464 30026 consensus_queue.cc:237] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:27.043363 27650 raft_consensus.cc:2932] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:27.046330 27257 raft_consensus.cc:1261] T c3a55662c6e346f0b73df334be4b14e3 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:27.046201 27386 raft_consensus.cc:2932] T d3a3473a93f14baaac48b9b7f47ac2a1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:27.045320 30287 raft_consensus.cc:2932] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:27.048754 2281 consensus_queue.cc:1035] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:27.053102 2944 raft_consensus.cc:2932] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:27.045714 2353 raft_consensus.cc:2932] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:27.057925 30289 raft_consensus.cc:2932] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:27.059407 2978 tablet_replica.cc:330] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:27.059675 2978 raft_consensus.cc:2229] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:27.064257 2980 ts_tablet_manager.cc:1888] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 08:18:27.064002 27110 catalog_manager.cc:5484] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 3 to 4, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:27.069147 2980 log.cc:1196] T c2776f857a5f47e9b8022639dc54757d P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/c2776f857a5f47e9b8022639dc54757d
I20240502 08:18:27.074816 27366 tablet_service.cc:1514] Processing DeleteTablet for tablet c3a55662c6e346f0b73df334be4b14e3 with delete_type TABLET_DATA_TOMBSTONED (TS 83e171a5e16c4008923f1d43fc5ce7b1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40868
I20240502 08:18:27.074932 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet c2776f857a5f47e9b8022639dc54757d (table rebalance_test_table_5 [id=5c5af31071894d4f803992d5173e0429]) successfully deleted
I20240502 08:18:27.075326 27257 raft_consensus.cc:1203] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:27.085574 27109 catalog_manager.cc:5484] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:27.085911 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3a55662c6e346f0b73df334be4b14e3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:27.089855 27255 raft_consensus.cc:2932] T c3a55662c6e346f0b73df334be4b14e3 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:27.092082 30290 raft_consensus.cc:1261] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:27.051853 27115 catalog_manager.cc:5484] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:27.093186 2281 consensus_queue.cc:1035] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:27.094727 30290 raft_consensus.cc:2932] T c3a55662c6e346f0b73df334be4b14e3 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:27.098531 30419 consensus_queue.cc:237] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:27.102420 30287 raft_consensus.cc:1261] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:27.101243 27386 raft_consensus.cc:1261] T d3a3473a93f14baaac48b9b7f47ac2a1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:27.102907 2965 consensus_queue.cc:1035] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:27.104429 2978 raft_consensus.cc:2258] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:27.105490 2962 consensus_queue.cc:1035] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:27.106700 2948 raft_consensus.cc:2932] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:27.108764 27387 raft_consensus.cc:2932] T d3a3473a93f14baaac48b9b7f47ac2a1 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:27.108530 2281 consensus_queue.cc:1035] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:27.110611 2566 consensus_queue.cc:1035] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:27.112145 2963 tablet_copy_client.cc:308] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.3:44469
I20240502 08:18:27.112783 27541 tablet_copy_service.cc:140] P b01e7c8a302d41078fa39cfef665ff59: Received BeginTabletCopySession request for tablet e80b9a98bdd041b1ac97af055951a92a from peer 49c5c8f70e494edd9fcda40cc0af6cc4 ({username='slave'} at 127.20.207.7:58763)
I20240502 08:18:27.112932 27541 tablet_copy_service.cc:161] P b01e7c8a302d41078fa39cfef665ff59: Beginning new tablet copy session on tablet e80b9a98bdd041b1ac97af055951a92a from peer 49c5c8f70e494edd9fcda40cc0af6cc4 at {username='slave'} at 127.20.207.7:58763: session id = 49c5c8f70e494edd9fcda40cc0af6cc4-e80b9a98bdd041b1ac97af055951a92a
I20240502 08:18:27.115702 27541 tablet_copy_source_session.cc:205] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:27.116387 2566 consensus_queue.cc:1035] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:27.117660 2963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e80b9a98bdd041b1ac97af055951a92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:27.100152 27258 raft_consensus.cc:2932] T 9edac369e0ae4c3c990bbb8dd845b27c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:27.126052 2976 ts_tablet_manager.cc:1888] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:18:27.126674 2976 log.cc:1196] T de4b822c4aca4f7f92b73cc9279ddec5 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/de4b822c4aca4f7f92b73cc9279ddec5
I20240502 08:18:27.127537 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet de4b822c4aca4f7f92b73cc9279ddec5 (table rebalancer_extra_table_26 [id=f18ddbbe6f0440debd82be537390727a]) successfully deleted
I20240502 08:18:27.126112 27649 raft_consensus.cc:2932] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:27.131430 27109 catalog_manager.cc:5484] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:27.128865 2985 ts_tablet_manager.cc:902] T 4165b86bc05e47f6a9803fb45ca01569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Initiating tablet copy from peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:27.133610 27518 raft_consensus.cc:2932] T ca8d1416a54344feae081ed7b19dd1de P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:27.134883 30287 raft_consensus.cc:2932] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:27.147107 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet 9edac369e0ae4c3c990bbb8dd845b27c with delete_type TABLET_DATA_TOMBSTONED (TS 49c5c8f70e494edd9fcda40cc0af6cc4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:27.147338 2978 ts_tablet_manager.cc:1875] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:27.147513 2976 tablet_replica.cc:330] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:27.147825 2976 raft_consensus.cc:2229] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:27.148113 2976 raft_consensus.cc:2258] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:27.149739 2976 ts_tablet_manager.cc:1875] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:27.149464 2353 raft_consensus.cc:2932] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:27.157053 27112 catalog_manager.cc:5484] T d3a3473a93f14baaac48b9b7f47ac2a1 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 3 to 4, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:27.157517 2963 tablet_copy_client.cc:783] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:27.157425 30287 consensus_queue.cc:237] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:27.157755 2963 tablet_copy_client.cc:647] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:27.157949 30423 raft_consensus.cc:2932] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:27.161320 27498 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a3473a93f14baaac48b9b7f47ac2a1 with delete_type TABLET_DATA_TOMBSTONED (TS b01e7c8a302d41078fa39cfef665ff59 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52776
I20240502 08:18:27.161620 2980 tablet_replica.cc:330] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59: stopping tablet replica
I20240502 08:18:27.161803 2980 raft_consensus.cc:2229] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:27.161993 2980 raft_consensus.cc:2258] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:27.162626 2980 ts_tablet_manager.cc:1875] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:27.166864 27650 raft_consensus.cc:1261] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:27.167526 2588 consensus_queue.cc:1035] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:27.173918 2987 tablet_replica.cc:330] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:18:27.174378 2980 ts_tablet_manager.cc:1888] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:18:27.174595 2980 log.cc:1196] T d3a3473a93f14baaac48b9b7f47ac2a1 P b01e7c8a302d41078fa39cfef665ff59: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-2/wal/wals/d3a3473a93f14baaac48b9b7f47ac2a1
I20240502 08:18:27.177788 27518 raft_consensus.cc:1261] T ca8d1416a54344feae081ed7b19dd1de P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:27.177821 2987 raft_consensus.cc:2229] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:27.178784 2987 raft_consensus.cc:2258] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:27.178757 2588 consensus_queue.cc:1035] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:27.181301 2948 raft_consensus.cc:2932] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:27.183104 2928 tablet_bootstrap.cc:492] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:27.183594 2928 tablet_bootstrap.cc:492] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6: Bootstrap complete.
I20240502 08:18:27.183820 2928 ts_tablet_manager.cc:1367] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.279s user 0.001s sys 0.006s
I20240502 08:18:27.183830 27103 catalog_manager.cc:4830] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): tablet d3a3473a93f14baaac48b9b7f47ac2a1 (table rebalancer_extra_table_46 [id=8d8448e41c7842ff8ddf670040a9aebd]) successfully deleted
I20240502 08:18:27.184754 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ca8d1416a54344feae081ed7b19dd1de with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:27.185770 27516 raft_consensus.cc:2932] T ca8d1416a54344feae081ed7b19dd1de P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:27.187799 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d3a3473a93f14baaac48b9b7f47ac2a1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:27.186547 2987 ts_tablet_manager.cc:1875] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:27.190932 27115 catalog_manager.cc:5484] T ca8d1416a54344feae081ed7b19dd1de P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:27.194391 30401 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8d1416a54344feae081ed7b19dd1de with delete_type TABLET_DATA_TOMBSTONED (TS 31a1e33dcdbf42f58ea72ce1c0afb0ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:27.197852 30418 raft_consensus.cc:2932] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:27.199972 2963 tablet_copy_client.cc:523] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:27.210764 2962 raft_consensus.cc:2932] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:27.220078 30290 raft_consensus.cc:2932] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:27.224552 2976 ts_tablet_manager.cc:1888] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:27.226125 30159 raft_consensus.cc:2932] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:27.227532 2976 log.cc:1196] T 9edac369e0ae4c3c990bbb8dd845b27c P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/9edac369e0ae4c3c990bbb8dd845b27c
I20240502 08:18:27.229696 27650 raft_consensus.cc:2932] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:27.232194 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet 9edac369e0ae4c3c990bbb8dd845b27c (table rebalancer_extra_table_42 [id=ae0aebcb27544669bbc3f8a4d649bb73]) successfully deleted
I20240502 08:18:27.237370 27115 catalog_manager.cc:5484] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 2 to 3, bf8ac71b15f740a0a928911360773442 (127.20.207.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:27.240010 2963 tablet_bootstrap.cc:492] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap starting.
I20240502 08:18:27.245198 2978 ts_tablet_manager.cc:1888] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:18:27.245373 2978 log.cc:1196] T 5e0313e85d444e098bfa35be89a3ad97 P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/5e0313e85d444e098bfa35be89a3ad97
I20240502 08:18:27.246090 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet 5e0313e85d444e098bfa35be89a3ad97 (table rebalance_test_table_9 [id=a65036830fa64a22b9f4efd6d5bd4f9f]) successfully deleted
I20240502 08:18:27.246031 30423 consensus_queue.cc:237] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:27.184556 2928 raft_consensus.cc:348] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:27.246320 2928 raft_consensus.cc:729] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: LEARNER
I20240502 08:18:27.247270 2928 consensus_queue.cc:260] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:27.248636 2965 raft_consensus.cc:1050] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad: attempting to promote NON_VOTER b01e7c8a302d41078fa39cfef665ff59 to VOTER
I20240502 08:18:27.249444 2965 consensus_queue.cc:237] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:27.251219 30287 raft_consensus.cc:1261] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:27.251677 2969 consensus_queue.cc:1035] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:27.254204 27650 raft_consensus.cc:1261] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:27.255082 2969 raft_consensus.cc:2932] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:27.255290 30290 raft_consensus.cc:2932] T 73548c1dd45f4c31bf3caf89b6d9ebac P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:27.259616 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 73548c1dd45f4c31bf3caf89b6d9ebac with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:27.263732 2928 ts_tablet_manager.cc:1398] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.080s user 0.001s sys 0.002s
I20240502 08:18:27.264349 2969 consensus_queue.cc:1035] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:27.265807 27649 raft_consensus.cc:2932] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:27.269767 27650 raft_consensus.cc:2932] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:27.278096 2997 tablet_replica.cc:330] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:27.278465 2997 raft_consensus.cc:2229] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:27.278805 2997 raft_consensus.cc:2258] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:27.281908 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 84cda20738d2452da166e31fc6da57a6-d122f68d36024614a017e8cc3a1224c1 received from {username='slave'} at 127.20.207.6:40325
I20240502 08:18:27.282054 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 84cda20738d2452da166e31fc6da57a6-d122f68d36024614a017e8cc3a1224c1 on tablet d122f68d36024614a017e8cc3a1224c1 with peer 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:27.283694 30287 consensus_queue.cc:237] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:27.284300 27115 catalog_manager.cc:5484] T 73548c1dd45f4c31bf3caf89b6d9ebac P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } }
I20240502 08:18:27.287971 30139 tablet_service.cc:1514] Processing DeleteTablet for tablet 73548c1dd45f4c31bf3caf89b6d9ebac with delete_type TABLET_DATA_TOMBSTONED (TS 84cda20738d2452da166e31fc6da57a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:27.299983 30156 raft_consensus.cc:1261] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:27.300338 30226 consensus_peers.cc:485] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecc002b16f4642159e199298acd38392. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:27.300542 30026 raft_consensus.cc:1261] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:27.300977 2589 consensus_queue.cc:1035] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:27.307724 2821 consensus_queue.cc:1035] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:27.307876 2963 tablet_bootstrap.cc:492] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:27.308524 2963 tablet_bootstrap.cc:492] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4: Bootstrap complete.
I20240502 08:18:27.308784 2963 ts_tablet_manager.cc:1367] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent bootstrapping tablet: real 0.069s user 0.006s sys 0.001s
I20240502 08:18:27.309517 2963 raft_consensus.cc:348] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:27.309710 2963 raft_consensus.cc:729] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Initialized, Role: LEARNER
I20240502 08:18:27.309917 2963 consensus_queue.cc:260] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:27.312377 2948 raft_consensus.cc:2932] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:27.317627 30026 raft_consensus.cc:2932] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:27.318439 3007 tablet_replica.cc:330] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:27.318670 3007 raft_consensus.cc:2229] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:27.318909 3007 raft_consensus.cc:2258] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:27.321259 3007 ts_tablet_manager.cc:1875] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:27.322211 27516 raft_consensus.cc:1261] T 4c78e068ec3a451c892d58174b3f837c P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:27.323168 30159 raft_consensus.cc:1261] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:27.324970 27650 raft_consensus.cc:1261] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:27.325260 2997 ts_tablet_manager.cc:1875] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:27.328685 27110 catalog_manager.cc:5484] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:27.329138 2965 consensus_queue.cc:1035] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:27.332707 2963 ts_tablet_manager.cc:1398] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240502 08:18:27.333290 2985 tablet_copy_client.cc:308] T 4165b86bc05e47f6a9803fb45ca01569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.6:38899
I20240502 08:18:27.333428 27541 tablet_copy_service.cc:342] P b01e7c8a302d41078fa39cfef665ff59: Request end of tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-e80b9a98bdd041b1ac97af055951a92a received from {username='slave'} at 127.20.207.7:58763
I20240502 08:18:27.333523 27541 tablet_copy_service.cc:433] P b01e7c8a302d41078fa39cfef665ff59: ending tablet copy session 49c5c8f70e494edd9fcda40cc0af6cc4-e80b9a98bdd041b1ac97af055951a92a on tablet e80b9a98bdd041b1ac97af055951a92a with peer 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:27.334666 2910 tablet_bootstrap.cc:492] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:27.335027 2965 consensus_queue.cc:1035] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:27.338378 30179 tablet_copy_service.cc:140] P 84cda20738d2452da166e31fc6da57a6: Received BeginTabletCopySession request for tablet 4165b86bc05e47f6a9803fb45ca01569 from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad ({username='slave'} at 127.20.207.8:52687)
I20240502 08:18:27.338554 30179 tablet_copy_service.cc:161] P 84cda20738d2452da166e31fc6da57a6: Beginning new tablet copy session on tablet 4165b86bc05e47f6a9803fb45ca01569 from peer 31a1e33dcdbf42f58ea72ce1c0afb0ad at {username='slave'} at 127.20.207.8:52687: session id = 31a1e33dcdbf42f58ea72ce1c0afb0ad-4165b86bc05e47f6a9803fb45ca01569
I20240502 08:18:27.341670 30179 tablet_copy_source_session.cc:205] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:27.342988 3007 ts_tablet_manager.cc:1888] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:27.343117 3007 log.cc:1196] T 73548c1dd45f4c31bf3caf89b6d9ebac P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/73548c1dd45f4c31bf3caf89b6d9ebac
I20240502 08:18:27.344105 2965 consensus_queue.cc:1035] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:27.345482 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet 73548c1dd45f4c31bf3caf89b6d9ebac (table rebalancer_extra_table_42 [id=ae0aebcb27544669bbc3f8a4d649bb73]) successfully deleted
I20240502 08:18:27.356303 2910 tablet_bootstrap.cc:492] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1: Bootstrap complete.
I20240502 08:18:27.365857 2997 ts_tablet_manager.cc:1888] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:27.366075 2997 log.cc:1196] T ca8d1416a54344feae081ed7b19dd1de P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/ca8d1416a54344feae081ed7b19dd1de
I20240502 08:18:27.366875 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet ca8d1416a54344feae081ed7b19dd1de (table rebalancer_extra_table_54 [id=a24d262efce245328528f2b42605f575]) successfully deleted
I20240502 08:18:27.387645 30157 raft_consensus.cc:2932] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:27.398236 2910 ts_tablet_manager.cc:1367] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent bootstrapping tablet: real 0.360s user 0.008s sys 0.002s
I20240502 08:18:27.402554 2910 raft_consensus.cc:348] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:27.403717 2985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4165b86bc05e47f6a9803fb45ca01569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:27.403483 2965 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:27.404279 30156 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:27.410638 2985 tablet_copy_client.cc:783] T 4165b86bc05e47f6a9803fb45ca01569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:27.410954 2985 tablet_copy_client.cc:647] T 4165b86bc05e47f6a9803fb45ca01569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:27.415331 2910 raft_consensus.cc:729] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Initialized, Role: LEARNER
I20240502 08:18:27.415874 27516 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:27.416206 27650 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:27.417830 27110 catalog_manager.cc:5484] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 5 to 6, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:27.414655 2987 ts_tablet_manager.cc:1888] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:27.420104 2910 consensus_queue.cc:260] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
I20240502 08:18:27.420624 2987 log.cc:1196] T c3a55662c6e346f0b73df334be4b14e3 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/c3a55662c6e346f0b73df334be4b14e3
I20240502 08:18:27.423233 30422 consensus_queue.cc:237] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:27.425264 2985 tablet_copy_client.cc:523] T 4165b86bc05e47f6a9803fb45ca01569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:27.428517 2910 ts_tablet_manager.cc:1398] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1: Time spent starting tablet: real 0.027s user 0.001s sys 0.000s
I20240502 08:18:27.428861 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet c3a55662c6e346f0b73df334be4b14e3 (table rebalancer_extra_table_44 [id=4aae824440954ca3be57248f5f1f6249]) successfully deleted
I20240502 08:18:27.436856 30179 tablet_copy_service.cc:342] P 84cda20738d2452da166e31fc6da57a6: Request end of tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-51201cc579394a2b819278a5943485a0 received from {username='slave'} at 127.20.207.2:56535
I20240502 08:18:27.437007 30179 tablet_copy_service.cc:433] P 84cda20738d2452da166e31fc6da57a6: ending tablet copy session 83e171a5e16c4008923f1d43fc5ce7b1-51201cc579394a2b819278a5943485a0 on tablet 51201cc579394a2b819278a5943485a0 with peer 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:27.438230 27518 raft_consensus.cc:1261] T 4c78e068ec3a451c892d58174b3f837c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:27.440563 2566 consensus_queue.cc:1035] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:27.441501 2985 tablet_bootstrap.cc:492] T 4165b86bc05e47f6a9803fb45ca01569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap starting.
I20240502 08:18:27.445443 2962 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:27.450961 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4c78e068ec3a451c892d58174b3f837c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:27.453168 27110 catalog_manager.cc:5484] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 6 to 7, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New cstate: current_term: 1 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:27.453608 30157 raft_consensus.cc:1261] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:27.455806 2965 consensus_queue.cc:1035] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:27.458148 27629 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c78e068ec3a451c892d58174b3f837c with delete_type TABLET_DATA_TOMBSTONED (TS bf8ac71b15f740a0a928911360773442 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44788
I20240502 08:18:27.456375 27516 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:27.461126 3016 tablet_replica.cc:330] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442: stopping tablet replica
I20240502 08:18:27.461447 3016 raft_consensus.cc:2229] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:27.461825 3016 raft_consensus.cc:2258] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:27.465190 3016 ts_tablet_manager.cc:1875] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:27.469247 30157 raft_consensus.cc:1203] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:27.473315 2985 tablet_bootstrap.cc:492] T 4165b86bc05e47f6a9803fb45ca01569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:27.475030 3016 ts_tablet_manager.cc:1888] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:18:27.475221 3016 log.cc:1196] T 4c78e068ec3a451c892d58174b3f837c P bf8ac71b15f740a0a928911360773442: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-3/wal/wals/4c78e068ec3a451c892d58174b3f837c
I20240502 08:18:27.476167 27103 catalog_manager.cc:4830] TS bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309): tablet 4c78e068ec3a451c892d58174b3f837c (table rebalancer_extra_table_36 [id=0f9f00e51ec94f5f9d43f80dbb78ff7a]) successfully deleted
I20240502 08:18:27.475970 2985 tablet_bootstrap.cc:492] T 4165b86bc05e47f6a9803fb45ca01569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Bootstrap complete.
I20240502 08:18:27.476554 30156 raft_consensus.cc:2932] T 4c78e068ec3a451c892d58174b3f837c P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bf8ac71b15f740a0a928911360773442 (127.20.207.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:27.477072 2985 ts_tablet_manager.cc:1367] T 4165b86bc05e47f6a9803fb45ca01569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.004s
I20240502 08:18:27.478869 2985 raft_consensus.cc:348] T 4165b86bc05e47f6a9803fb45ca01569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:27.479193 2985 raft_consensus.cc:729] T 4165b86bc05e47f6a9803fb45ca01569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Initialized, Role: LEARNER
I20240502 08:18:27.479542 2985 consensus_queue.cc:260] T 4165b86bc05e47f6a9803fb45ca01569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } }
I20240502 08:18:27.482518 2985 ts_tablet_manager.cc:1398] T 4165b86bc05e47f6a9803fb45ca01569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:18:27.483621 30179 tablet_copy_service.cc:342] P 84cda20738d2452da166e31fc6da57a6: Request end of tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-4165b86bc05e47f6a9803fb45ca01569 received from {username='slave'} at 127.20.207.8:52687
I20240502 08:18:27.483742 30179 tablet_copy_service.cc:433] P 84cda20738d2452da166e31fc6da57a6: ending tablet copy session 31a1e33dcdbf42f58ea72ce1c0afb0ad-4165b86bc05e47f6a9803fb45ca01569 on tablet 4165b86bc05e47f6a9803fb45ca01569 with peer 31a1e33dcdbf42f58ea72ce1c0afb0ad
I20240502 08:18:27.506512 27389 consensus_queue.cc:237] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:27.508584 30026 raft_consensus.cc:1261] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:27.508769 27516 raft_consensus.cc:1261] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:18:27.508886 27326 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): Couldn't send request to peer 84cda20738d2452da166e31fc6da57a6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6cf739b314d84db08bc4cb32b0e5b0f4. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:27.509093 2513 consensus_queue.cc:1035] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:27.509378 3023 consensus_queue.cc:1035] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:27.514840 2758 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:27.515542 30026 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:27.522522 27114 catalog_manager.cc:5484] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:27.523736 27516 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } } }
I20240502 08:18:27.625072 30026 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2e9b191ce9234c028b990efd74a59e2e"
dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47292
I20240502 08:18:27.625293 30026 raft_consensus.cc:595] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:18:27.659962 27389 raft_consensus.cc:1203] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:27.665553 2513 raft_consensus.cc:1050] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1: attempting to promote NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 to VOTER
I20240502 08:18:27.666338 2513 consensus_queue.cc:237] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:27.675211 27516 raft_consensus.cc:1261] T 0ed6b8adc62f471696cbf0330524e353 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:27.675979 2758 consensus_queue.cc:1035] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:27.678792 27257 raft_consensus.cc:1261] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:27.678826 30156 raft_consensus.cc:1261] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:27.679546 3023 consensus_queue.cc:1035] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:27.680213 3023 consensus_queue.cc:1035] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:27.691243 3023 raft_consensus.cc:2932] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:27.697369 27257 raft_consensus.cc:2932] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:27.706036 27110 catalog_manager.cc:5484] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:27.718545 30156 raft_consensus.cc:2932] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:27.719580 27516 raft_consensus.cc:2932] T 0ed6b8adc62f471696cbf0330524e353 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:27.728555 27387 consensus_queue.cc:237] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:27.730175 30422 raft_consensus.cc:1203] T 4165b86bc05e47f6a9803fb45ca01569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:27.730904 27518 raft_consensus.cc:1261] T 0ed6b8adc62f471696cbf0330524e353 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:27.731258 27258 raft_consensus.cc:1261] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:27.731645 2513 consensus_queue.cc:1035] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:27.732172 2513 consensus_queue.cc:1035] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:27.734755 3023 raft_consensus.cc:2932] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:27.738896 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0ed6b8adc62f471696cbf0330524e353 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:27.742868 27257 raft_consensus.cc:2932] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:27.749648 27114 catalog_manager.cc:5484] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:27.759158 27516 raft_consensus.cc:2932] T 0ed6b8adc62f471696cbf0330524e353 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } }
I20240502 08:18:27.764091 30139 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ed6b8adc62f471696cbf0330524e353 with delete_type TABLET_DATA_TOMBSTONED (TS 84cda20738d2452da166e31fc6da57a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:27.764407 3007 tablet_replica.cc:330] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:27.764592 3007 raft_consensus.cc:2229] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:27.764798 3007 raft_consensus.cc:2258] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:27.765661 3007 ts_tablet_manager.cc:1875] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:27.771780 3007 ts_tablet_manager.cc:1888] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:27.772732 3007 log.cc:1196] T 0ed6b8adc62f471696cbf0330524e353 P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/0ed6b8adc62f471696cbf0330524e353
I20240502 08:18:27.773548 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet 0ed6b8adc62f471696cbf0330524e353 (table rebalancer_extra_table_38 [id=beadc4ad4dcd49e19ca5461ae00bb35f]) successfully deleted
I20240502 08:18:27.791885 30290 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f1f5650965e841ad98d857ca32299037"
dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36368
I20240502 08:18:27.792117 30290 raft_consensus.cc:595] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:18:27.797942 2821 raft_consensus.cc:979] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4: : Instructing follower 1f0bf4bc3c5146539146b825b048bff3 to start an election
I20240502 08:18:27.798130 2821 raft_consensus.cc:1067] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Signalling peer 1f0bf4bc3c5146539146b825b048bff3 to start an election
I20240502 08:18:27.799054 30026 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f1f5650965e841ad98d857ca32299037"
dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
from {username='slave'} at 127.20.207.7:60733
I20240502 08:18:27.799263 30026 raft_consensus.cc:482] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:27.799372 30026 raft_consensus.cc:3037] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:27.802237 3101 ts_tablet_manager.cc:902] T ecc002b16f4642159e199298acd38392 P b01e7c8a302d41078fa39cfef665ff59: Initiating tablet copy from peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:27.802148 30026 raft_consensus.cc:504] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:27.803944 30156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f5650965e841ad98d857ca32299037" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:27.804172 30156 raft_consensus.cc:3037] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:27.804239 30026 leader_election.cc:288] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:27.806612 30156 raft_consensus.cc:2454] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 2.
I20240502 08:18:27.808737 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f5650965e841ad98d857ca32299037" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:27.808992 30422 raft_consensus.cc:3037] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:27.811349 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f5650965e841ad98d857ca32299037" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4"
I20240502 08:18:27.811471 30422 raft_consensus.cc:2454] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 2.
I20240502 08:18:27.811554 30290 raft_consensus.cc:3032] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:27.811637 30290 raft_consensus.cc:729] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49c5c8f70e494edd9fcda40cc0af6cc4, State: Running, Role: LEADER
I20240502 08:18:27.811903 30290 consensus_queue.cc:260] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:27.812320 30290 raft_consensus.cc:3037] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:27.812443 30423 raft_consensus.cc:1226] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Rejecting Update request from peer 49c5c8f70e494edd9fcda40cc0af6cc4 for earlier term 1. Current term is 2. Ops: []
I20240502 08:18:27.813231 29961 leader_election.cc:302] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:27.813496 2281 raft_consensus.cc:2781] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:27.813603 2281 raft_consensus.cc:686] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:18:27.813858 2281 consensus_queue.cc:237] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:27.815321 30290 raft_consensus.cc:2454] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 2.
I20240502 08:18:27.821395 27110 catalog_manager.cc:5484] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 1 to 2, leader changed from 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 2 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:27.845865 2281 raft_consensus.cc:979] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3: : Instructing follower 83e171a5e16c4008923f1d43fc5ce7b1 to start an election
I20240502 08:18:27.846117 2281 raft_consensus.cc:1067] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Signalling peer 83e171a5e16c4008923f1d43fc5ce7b1 to start an election
I20240502 08:18:27.847748 27387 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2e9b191ce9234c028b990efd74a59e2e"
dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1"
from {username='slave'} at 127.20.207.5:49101
I20240502 08:18:27.848026 27387 raft_consensus.cc:482] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:27.848135 27387 raft_consensus.cc:3037] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:27.850311 30290 raft_consensus.cc:1203] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:27.851465 27387 raft_consensus.cc:504] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:27.852481 30422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9b191ce9234c028b990efd74a59e2e" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:27.852713 30422 raft_consensus.cc:3037] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:27.856518 30422 raft_consensus.cc:2454] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 2.
I20240502 08:18:27.860005 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9b191ce9234c028b990efd74a59e2e" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:27.860225 30025 raft_consensus.cc:3032] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:27.860327 30025 raft_consensus.cc:729] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:18:27.860647 30025 consensus_queue.cc:260] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:27.861092 30025 raft_consensus.cc:3037] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:27.862409 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9b191ce9234c028b990efd74a59e2e" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:27.862599 27257 raft_consensus.cc:3037] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:27.864449 30025 raft_consensus.cc:2454] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 2.
I20240502 08:18:27.865224 3101 tablet_copy_client.cc:308] T ecc002b16f4642159e199298acd38392 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.7:40343
I20240502 08:18:27.865800 30310 tablet_copy_service.cc:140] P 49c5c8f70e494edd9fcda40cc0af6cc4: Received BeginTabletCopySession request for tablet ecc002b16f4642159e199298acd38392 from peer b01e7c8a302d41078fa39cfef665ff59 ({username='slave'} at 127.20.207.3:49869)
I20240502 08:18:27.865885 27257 raft_consensus.cc:2454] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 2.
I20240502 08:18:27.865947 30310 tablet_copy_service.cc:161] P 49c5c8f70e494edd9fcda40cc0af6cc4: Beginning new tablet copy session on tablet ecc002b16f4642159e199298acd38392 from peer b01e7c8a302d41078fa39cfef665ff59 at {username='slave'} at 127.20.207.3:49869: session id = b01e7c8a302d41078fa39cfef665ff59-ecc002b16f4642159e199298acd38392
I20240502 08:18:27.866253 27323 leader_election.cc:302] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 31a1e33dcdbf42f58ea72ce1c0afb0ad, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
I20240502 08:18:27.866497 3087 raft_consensus.cc:2781] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:27.866597 3087 raft_consensus.cc:686] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 2 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:18:27.866922 3087 consensus_queue.cc:237] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:27.868495 30310 tablet_copy_source_session.cc:205] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:27.868867 27387 leader_election.cc:288] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 2 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:27.870982 3101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc002b16f4642159e199298acd38392. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:27.871354 27110 catalog_manager.cc:5484] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 1 to 2, leader changed from 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 2 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:27.876879 3101 tablet_copy_client.cc:783] T ecc002b16f4642159e199298acd38392 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:27.877101 3101 tablet_copy_client.cc:647] T ecc002b16f4642159e199298acd38392 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:27.878685 3101 tablet_copy_client.cc:523] T ecc002b16f4642159e199298acd38392 P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:27.886531 3101 tablet_bootstrap.cc:492] T ecc002b16f4642159e199298acd38392 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:27.900486 3101 tablet_bootstrap.cc:492] T ecc002b16f4642159e199298acd38392 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:27.900933 3101 tablet_bootstrap.cc:492] T ecc002b16f4642159e199298acd38392 P b01e7c8a302d41078fa39cfef665ff59: Bootstrap complete.
I20240502 08:18:27.901209 3101 ts_tablet_manager.cc:1367] T ecc002b16f4642159e199298acd38392 P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.008s
I20240502 08:18:27.901896 3101 raft_consensus.cc:348] T ecc002b16f4642159e199298acd38392 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:27.902149 3101 raft_consensus.cc:729] T ecc002b16f4642159e199298acd38392 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: LEARNER
I20240502 08:18:27.902356 3101 consensus_queue.cc:260] T ecc002b16f4642159e199298acd38392 P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:27.902905 3101 ts_tablet_manager.cc:1398] T ecc002b16f4642159e199298acd38392 P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:27.903571 30310 tablet_copy_service.cc:342] P 49c5c8f70e494edd9fcda40cc0af6cc4: Request end of tablet copy session b01e7c8a302d41078fa39cfef665ff59-ecc002b16f4642159e199298acd38392 received from {username='slave'} at 127.20.207.3:49869
I20240502 08:18:27.903676 30310 tablet_copy_service.cc:433] P 49c5c8f70e494edd9fcda40cc0af6cc4: ending tablet copy session b01e7c8a302d41078fa39cfef665ff59-ecc002b16f4642159e199298acd38392 on tablet ecc002b16f4642159e199298acd38392 with peer b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:27.951752 2800 raft_consensus.cc:1050] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 84cda20738d2452da166e31fc6da57a6 to VOTER
I20240502 08:18:27.952594 2800 consensus_queue.cc:237] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:27.958570 30026 raft_consensus.cc:1261] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:27.959642 2814 consensus_queue.cc:1035] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:27.959765 30422 raft_consensus.cc:1261] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:27.961794 2814 consensus_queue.cc:1035] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:27.963356 30156 raft_consensus.cc:1261] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:27.964385 2814 consensus_queue.cc:1035] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:27.972771 3134 ts_tablet_manager.cc:902] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6: Initiating tablet copy from peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:27.973908 2814 raft_consensus.cc:2932] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:27.978353 30422 raft_consensus.cc:2932] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:27.991338 27114 catalog_manager.cc:5484] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index 2 to 3, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:28.012980 30026 raft_consensus.cc:2932] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:28.022657 30156 raft_consensus.cc:2932] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:28.042912 27518 consensus_queue.cc:237] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:28.044813 30156 raft_consensus.cc:1261] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:28.045338 30026 raft_consensus.cc:1261] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:28.045851 3129 consensus_queue.cc:1035] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.046909 2813 consensus_queue.cc:1035] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.050968 3134 tablet_copy_client.cc:308] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.2:46291
I20240502 08:18:28.050844 2813 raft_consensus.cc:2932] T d122f68d36024614a017e8cc3a1224c1 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:28.051497 30026 raft_consensus.cc:2932] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:28.054517 27410 tablet_copy_service.cc:140] P 83e171a5e16c4008923f1d43fc5ce7b1: Received BeginTabletCopySession request for tablet 6cf739b314d84db08bc4cb32b0e5b0f4 from peer 84cda20738d2452da166e31fc6da57a6 ({username='slave'} at 127.20.207.6:49833)
I20240502 08:18:28.054702 27410 tablet_copy_service.cc:161] P 83e171a5e16c4008923f1d43fc5ce7b1: Beginning new tablet copy session on tablet 6cf739b314d84db08bc4cb32b0e5b0f4 from peer 84cda20738d2452da166e31fc6da57a6 at {username='slave'} at 127.20.207.6:49833: session id = 84cda20738d2452da166e31fc6da57a6-6cf739b314d84db08bc4cb32b0e5b0f4
I20240502 08:18:28.057687 27110 catalog_manager.cc:5484] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:28.066155 27410 tablet_copy_source_session.cc:205] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:28.066427 30401 tablet_service.cc:1514] Processing DeleteTablet for tablet d122f68d36024614a017e8cc3a1224c1 with delete_type TABLET_DATA_TOMBSTONED (TS 31a1e33dcdbf42f58ea72ce1c0afb0ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:28.067641 3134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf739b314d84db08bc4cb32b0e5b0f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:28.069046 30156 raft_consensus.cc:2932] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:28.075577 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d122f68d36024614a017e8cc3a1224c1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:28.076260 3134 tablet_copy_client.cc:783] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:28.076560 3134 tablet_copy_client.cc:647] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:28.080969 3134 tablet_copy_client.cc:523] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:28.082393 3148 tablet_replica.cc:330] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:28.082670 3148 raft_consensus.cc:2229] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:28.083778 3134 tablet_bootstrap.cc:492] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6: Bootstrap starting.
I20240502 08:18:28.084270 3148 raft_consensus.cc:2258] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:28.085589 3148 ts_tablet_manager.cc:1875] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:28.102241 3148 ts_tablet_manager.cc:1888] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:28.102490 3148 log.cc:1196] T d122f68d36024614a017e8cc3a1224c1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/d122f68d36024614a017e8cc3a1224c1
I20240502 08:18:28.103571 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet d122f68d36024614a017e8cc3a1224c1 (table rebalancer_extra_table_46 [id=8d8448e41c7842ff8ddf670040a9aebd]) successfully deleted
I20240502 08:18:28.124378 30290 consensus_queue.cc:237] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:28.126884 27649 raft_consensus.cc:1261] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:28.127137 27387 raft_consensus.cc:1261] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:28.127602 2589 consensus_queue.cc:1035] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:28.127864 2948 consensus_queue.cc:1035] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:18:28.128700 30225 consensus_peers.cc:485] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 -> Peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): Couldn't send request to peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1f416f090284b8d92d53662cc7a3b97. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:28.130582 2589 raft_consensus.cc:2932] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:28.134775 27387 raft_consensus.cc:2932] T f1f416f090284b8d92d53662cc7a3b97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:28.138051 27110 catalog_manager.cc:5484] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 4 to 5, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:28.145143 3134 tablet_bootstrap.cc:492] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:28.146168 3134 tablet_bootstrap.cc:492] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6: Bootstrap complete.
I20240502 08:18:28.146440 3134 ts_tablet_manager.cc:1367] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6: Time spent bootstrapping tablet: real 0.063s user 0.044s sys 0.004s
I20240502 08:18:28.147768 27649 raft_consensus.cc:2932] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } } }
I20240502 08:18:28.149250 3134 raft_consensus.cc:348] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:28.149605 3134 raft_consensus.cc:729] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Initialized, Role: LEARNER
I20240502 08:18:28.149855 3134 consensus_queue.cc:260] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: NON_VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: true } }
I20240502 08:18:28.153347 3134 ts_tablet_manager.cc:1398] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:18:28.154027 27410 tablet_copy_service.cc:342] P 83e171a5e16c4008923f1d43fc5ce7b1: Request end of tablet copy session 84cda20738d2452da166e31fc6da57a6-6cf739b314d84db08bc4cb32b0e5b0f4 received from {username='slave'} at 127.20.207.6:49833
I20240502 08:18:28.154125 27410 tablet_copy_service.cc:433] P 83e171a5e16c4008923f1d43fc5ce7b1: ending tablet copy session 84cda20738d2452da166e31fc6da57a6-6cf739b314d84db08bc4cb32b0e5b0f4 on tablet 6cf739b314d84db08bc4cb32b0e5b0f4 with peer 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:28.167412 2954 raft_consensus.cc:1050] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6: attempting to promote NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 to VOTER
I20240502 08:18:28.168190 2954 consensus_queue.cc:237] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:28.170563 30156 consensus_queue.cc:237] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:28.172225 30422 raft_consensus.cc:1261] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:28.172363 27387 raft_consensus.cc:1261] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEARNER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:28.172961 2520 consensus_queue.cc:1035] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:28.173837 2960 consensus_queue.cc:1035] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:28.176491 30422 raft_consensus.cc:1261] T a52fe5caeb364b868a1965633a7da09e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:28.176663 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a52fe5caeb364b868a1965633a7da09e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:28.176709 2960 raft_consensus.cc:2932] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:28.177273 2954 consensus_queue.cc:1035] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:28.180048 2954 raft_consensus.cc:2932] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:28.180449 27389 raft_consensus.cc:1261] T a52fe5caeb364b868a1965633a7da09e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:28.180836 27387 raft_consensus.cc:2932] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:28.180939 30422 raft_consensus.cc:2932] T a52fe5caeb364b868a1965633a7da09e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:28.186131 27649 raft_consensus.cc:1261] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:28.186568 2954 consensus_queue.cc:1035] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:28.188556 27110 catalog_manager.cc:5484] T a52fe5caeb364b868a1965633a7da09e P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:28.189373 27115 catalog_manager.cc:5484] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:28.212006 27387 raft_consensus.cc:2932] T a52fe5caeb364b868a1965633a7da09e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:28.219398 30422 raft_consensus.cc:2932] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:28.221141 2954 consensus_queue.cc:1035] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:28.239542 3164 raft_consensus.cc:1050] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6: attempting to promote NON_VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad to VOTER
I20240502 08:18:28.240367 3164 consensus_queue.cc:237] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:28.238312 27649 raft_consensus.cc:2932] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:28.243718 30422 raft_consensus.cc:1261] T 4165b86bc05e47f6a9803fb45ca01569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 LEARNER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:28.244454 27516 raft_consensus.cc:1261] T 4165b86bc05e47f6a9803fb45ca01569 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:28.244832 27387 raft_consensus.cc:1261] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:28.245095 2960 consensus_queue.cc:1035] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:28.245340 3170 consensus_queue.cc:1035] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240502 08:18:28.247826 3164 consensus_queue.cc:1035] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:28.253330 30423 raft_consensus.cc:1261] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:18:28.254017 2943 consensus_queue.cc:1035] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.262037 30290 raft_consensus.cc:1261] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:18:28.268573 3055 consensus_queue.cc:1035] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.270614 3170 raft_consensus.cc:2932] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:28.271281 30026 consensus_queue.cc:237] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:28.275172 27520 raft_consensus.cc:2932] T 4165b86bc05e47f6a9803fb45ca01569 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:28.275627 27649 raft_consensus.cc:1261] T cef42b07900c4b4e8589f284c011c300 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:28.275888 30287 raft_consensus.cc:1261] T cef42b07900c4b4e8589f284c011c300 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:28.276422 2943 consensus_queue.cc:1035] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:28.279284 2943 consensus_queue.cc:1035] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:28.283133 30422 raft_consensus.cc:2932] T 4165b86bc05e47f6a9803fb45ca01569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:28.284152 27110 catalog_manager.cc:5484] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 2 to 3, 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:28.287009 29964 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cef42b07900c4b4e8589f284c011c300. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:28.296651 30156 consensus_queue.cc:237] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:28.299288 27516 raft_consensus.cc:1261] T 4165b86bc05e47f6a9803fb45ca01569 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:28.299036 2281 raft_consensus.cc:2932] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:28.299458 30289 raft_consensus.cc:2932] T cef42b07900c4b4e8589f284c011c300 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:28.300217 2954 consensus_queue.cc:1035] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.310127 2954 raft_consensus.cc:2932] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:28.312105 30422 raft_consensus.cc:1261] T 4165b86bc05e47f6a9803fb45ca01569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:28.312690 3164 consensus_queue.cc:1035] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.313728 27516 raft_consensus.cc:2932] T 4165b86bc05e47f6a9803fb45ca01569 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:28.314388 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4165b86bc05e47f6a9803fb45ca01569 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:28.316634 27115 catalog_manager.cc:5484] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New cstate: current_term: 1 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:28.317723 27110 catalog_manager.cc:5484] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index 3 to 4, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:28.320199 30422 raft_consensus.cc:2932] T 4165b86bc05e47f6a9803fb45ca01569 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } }
I20240502 08:18:28.324234 27366 tablet_service.cc:1514] Processing DeleteTablet for tablet 4165b86bc05e47f6a9803fb45ca01569 with delete_type TABLET_DATA_TOMBSTONED (TS 83e171a5e16c4008923f1d43fc5ce7b1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40868
I20240502 08:18:28.317271 27652 raft_consensus.cc:2932] T cef42b07900c4b4e8589f284c011c300 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } } }
I20240502 08:18:28.363502 27516 raft_consensus.cc:1203] T ecc002b16f4642159e199298acd38392 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:28.372880 3190 tablet_replica.cc:330] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1: stopping tablet replica
I20240502 08:18:28.373183 3190 raft_consensus.cc:2229] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:28.373426 3190 pending_rounds.cc:62] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1: Trying to abort 1 pending ops.
I20240502 08:18:28.373471 3190 pending_rounds.cc:68] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023156872150470656 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "4165b86bc05e47f6a9803fb45ca01569" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: NON_VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } } }
I20240502 08:18:28.373944 3190 raft_consensus.cc:2894] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240502 08:18:28.374125 3190 raft_consensus.cc:2866] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 08:18:28.374217 3190 raft_consensus.cc:2258] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:18:28.375597 3181 consensus_queue.cc:1159] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" status { last_received { term: 1 index: 3 } last_committed_idx: 2 last_received_current_leader { term: 1 index: 3 } }
I20240502 08:18:28.390153 30157 raft_consensus.cc:1203] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:18:28.394776 3190 ts_tablet_manager.cc:1875] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:28.406458 30156 raft_consensus.cc:1261] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:18:28.407132 3055 consensus_queue.cc:1035] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.407339 27257 raft_consensus.cc:1261] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:18:28.410609 3196 consensus_queue.cc:1035] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.420892 3190 ts_tablet_manager.cc:1888] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:28.424933 3190 log.cc:1196] T 4165b86bc05e47f6a9803fb45ca01569 P 83e171a5e16c4008923f1d43fc5ce7b1: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-1/wal/wals/4165b86bc05e47f6a9803fb45ca01569
I20240502 08:18:28.434067 27102 catalog_manager.cc:4830] TS 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): tablet 4165b86bc05e47f6a9803fb45ca01569 (table rebalancer_extra_table_38 [id=beadc4ad4dcd49e19ca5461ae00bb35f]) successfully deleted
I20240502 08:18:28.434983 2800 raft_consensus.cc:1050] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59: attempting to promote NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 to VOTER
I20240502 08:18:28.435880 2800 consensus_queue.cc:237] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:28.449195 30289 raft_consensus.cc:1261] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEARNER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:28.449872 2814 consensus_queue.cc:1035] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:28.453130 27255 raft_consensus.cc:1261] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:28.453743 2800 consensus_queue.cc:1035] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:28.455991 30026 raft_consensus.cc:1261] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:28.456463 30025 raft_consensus.cc:1261] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:18:28.456990 3195 consensus_queue.cc:1035] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.456560 2800 consensus_queue.cc:1035] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 08:18:28.466475 30156 consensus_queue.cc:237] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } }
I20240502 08:18:28.475724 30423 raft_consensus.cc:1261] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:18:28.476431 3195 consensus_queue.cc:1035] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.477519 2814 raft_consensus.cc:2932] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:28.484263 30026 raft_consensus.cc:2932] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:28.485726 27257 raft_consensus.cc:2932] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:28.496417 30289 raft_consensus.cc:2932] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:28.497357 27114 catalog_manager.cc:5484] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 2 to 3, 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { replace: true } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:28.521860 27516 consensus_queue.cc:237] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:28.547971 27387 consensus_queue.cc:237] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:28.549845 27255 raft_consensus.cc:1261] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:18:28.550417 3195 consensus_queue.cc:1035] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:28.551347 30422 raft_consensus.cc:1261] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:18:28.551910 3195 consensus_queue.cc:1035] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:28.553328 3196 raft_consensus.cc:2932] T 2e9b191ce9234c028b990efd74a59e2e P 83e171a5e16c4008923f1d43fc5ce7b1 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:28.553972 27255 raft_consensus.cc:2932] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
W20240502 08:18:28.557044 27453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.5:37815, user_credentials={real_user=slave}} blocked reactor thread for 62996.3us
I20240502 08:18:28.559937 30287 raft_consensus.cc:1261] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:28.560334 27257 raft_consensus.cc:1261] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:28.560690 30418 raft_consensus.cc:1261] T c30720c1b245438db3eb4bea2b8bd930 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:28.560986 3170 consensus_queue.cc:1035] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:28.561327 2954 consensus_queue.cc:1035] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:28.563226 30093 consensus_peers.cc:485] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c30720c1b245438db3eb4bea2b8bd930. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:28.563647 30026 raft_consensus.cc:1261] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer b01e7c8a302d41078fa39cfef665ff59: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:28.563464 27115 catalog_manager.cc:5484] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index 3 to 5, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New cstate: current_term: 2 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:28.565351 3129 consensus_queue.cc:1035] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.567955 30007 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e9b191ce9234c028b990efd74a59e2e with delete_type TABLET_DATA_TOMBSTONED (TS 1f0bf4bc3c5146539146b825b048bff3 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58374
I20240502 08:18:28.568943 3129 consensus_queue.cc:1035] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.571590 3211 raft_consensus.cc:2932] T e80b9a98bdd041b1ac97af055951a92a P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:28.572561 30290 raft_consensus.cc:2932] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:28.579349 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e80b9a98bdd041b1ac97af055951a92a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:28.582221 27115 catalog_manager.cc:5484] T e80b9a98bdd041b1ac97af055951a92a P 49c5c8f70e494edd9fcda40cc0af6cc4 reported cstate change: config changed from index 3 to 4, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New cstate: current_term: 1 leader_uuid: "b01e7c8a302d41078fa39cfef665ff59" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:28.584527 30026 raft_consensus.cc:2932] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } }
I20240502 08:18:28.595469 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet e80b9a98bdd041b1ac97af055951a92a with delete_type TABLET_DATA_TOMBSTONED (TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39166
I20240502 08:18:28.598071 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2e9b191ce9234c028b990efd74a59e2e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:28.598158 30422 raft_consensus.cc:2932] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:28.601907 2954 raft_consensus.cc:2932] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:28.608525 27112 catalog_manager.cc:5484] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:28.613704 3219 tablet_replica.cc:330] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: stopping tablet replica
I20240502 08:18:28.614607 3219 raft_consensus.cc:2229] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:28.614886 3219 raft_consensus.cc:2258] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:28.616333 3216 tablet_replica.cc:330] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3: stopping tablet replica
I20240502 08:18:28.616529 3216 raft_consensus.cc:2229] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:28.616716 3216 pending_rounds.cc:62] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3: Trying to abort 1 pending ops.
I20240502 08:18:28.616770 3216 pending_rounds.cc:68] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3: Aborting op as it isn't in flight: id { term: 2 index: 4 } timestamp: 7023156870626430976 op_type: NO_OP noop_request { }
I20240502 08:18:28.616906 3216 raft_consensus.cc:2866] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:18:28.617003 3216 raft_consensus.cc:2258] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:28.618892 3219 ts_tablet_manager.cc:1875] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:28.620997 3216 ts_tablet_manager.cc:1875] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:28.623649 30418 raft_consensus.cc:2932] T c30720c1b245438db3eb4bea2b8bd930 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:28.626430 3218 ts_tablet_manager.cc:902] T f1f416f090284b8d92d53662cc7a3b97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Initiating tablet copy from peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:28.631788 3216 ts_tablet_manager.cc:1888] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 08:18:28.632062 3216 log.cc:1196] T 2e9b191ce9234c028b990efd74a59e2e P 1f0bf4bc3c5146539146b825b048bff3: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-4/wal/wals/2e9b191ce9234c028b990efd74a59e2e
I20240502 08:18:28.633054 27100 catalog_manager.cc:4830] TS 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815): tablet 2e9b191ce9234c028b990efd74a59e2e (table rebalancer_extra_table_40 [id=50fcb176b4c44d52a482856f676a46d5]) successfully deleted
I20240502 08:18:28.654906 30156 consensus_queue.cc:237] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:28.656042 3219 ts_tablet_manager.cc:1888] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:28.656189 3219 log.cc:1196] T e80b9a98bdd041b1ac97af055951a92a P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-0/wal/wals/e80b9a98bdd041b1ac97af055951a92a
I20240502 08:18:28.657793 27652 raft_consensus.cc:1261] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:28.658367 27387 raft_consensus.cc:1261] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:28.658344 3164 consensus_queue.cc:1035] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.658917 3164 consensus_queue.cc:1035] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.659173 27102 catalog_manager.cc:4830] TS 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505): tablet e80b9a98bdd041b1ac97af055951a92a (table rebalancer_extra_table_48 [id=327af2c89b2f41028a116c0587a226ae]) successfully deleted
I20240502 08:18:28.661451 2954 raft_consensus.cc:2932] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:28.663339 27652 raft_consensus.cc:2932] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:28.665522 27387 raft_consensus.cc:2932] T 51201cc579394a2b819278a5943485a0 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:28.667284 27103 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 51201cc579394a2b819278a5943485a0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:28.671600 27114 catalog_manager.cc:5484] T 51201cc579394a2b819278a5943485a0 P bf8ac71b15f740a0a928911360773442 reported cstate change: config changed from index 3 to 4, VOTER 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8) evicted. New cstate: current_term: 1 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } } }
I20240502 08:18:28.676529 30401 tablet_service.cc:1514] Processing DeleteTablet for tablet 51201cc579394a2b819278a5943485a0 with delete_type TABLET_DATA_TOMBSTONED (TS 31a1e33dcdbf42f58ea72ce1c0afb0ad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56694
I20240502 08:18:28.684599 27257 raft_consensus.cc:2932] T c30720c1b245438db3eb4bea2b8bd930 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: NON_VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: true } } }
I20240502 08:18:28.687112 3217 ts_tablet_manager.cc:902] T a52fe5caeb364b868a1965633a7da09e P b01e7c8a302d41078fa39cfef665ff59: Initiating tablet copy from peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:28.692185 3217 tablet_copy_client.cc:308] T a52fe5caeb364b868a1965633a7da09e P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.6:38899
I20240502 08:18:28.692766 30179 tablet_copy_service.cc:140] P 84cda20738d2452da166e31fc6da57a6: Received BeginTabletCopySession request for tablet a52fe5caeb364b868a1965633a7da09e from peer b01e7c8a302d41078fa39cfef665ff59 ({username='slave'} at 127.20.207.3:35877)
I20240502 08:18:28.692948 30179 tablet_copy_service.cc:161] P 84cda20738d2452da166e31fc6da57a6: Beginning new tablet copy session on tablet a52fe5caeb364b868a1965633a7da09e from peer b01e7c8a302d41078fa39cfef665ff59 at {username='slave'} at 127.20.207.3:35877: session id = b01e7c8a302d41078fa39cfef665ff59-a52fe5caeb364b868a1965633a7da09e
I20240502 08:18:28.695804 30179 tablet_copy_source_session.cc:205] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:28.697072 3217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52fe5caeb364b868a1965633a7da09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:28.712432 3217 tablet_copy_client.cc:783] T a52fe5caeb364b868a1965633a7da09e P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:28.712677 3217 tablet_copy_client.cc:647] T a52fe5caeb364b868a1965633a7da09e P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:28.714969 3217 tablet_copy_client.cc:523] T a52fe5caeb364b868a1965633a7da09e P b01e7c8a302d41078fa39cfef665ff59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:28.717473 3218 tablet_copy_client.cc:308] T f1f416f090284b8d92d53662cc7a3b97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Beginning tablet copy session from remote peer at address 127.20.207.7:40343
I20240502 08:18:28.721661 3217 tablet_bootstrap.cc:492] T a52fe5caeb364b868a1965633a7da09e P b01e7c8a302d41078fa39cfef665ff59: Bootstrap starting.
I20240502 08:18:28.738271 30310 tablet_copy_service.cc:140] P 49c5c8f70e494edd9fcda40cc0af6cc4: Received BeginTabletCopySession request for tablet f1f416f090284b8d92d53662cc7a3b97 from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 ({username='slave'} at 127.20.207.1:49483)
I20240502 08:18:28.738451 30310 tablet_copy_service.cc:161] P 49c5c8f70e494edd9fcda40cc0af6cc4: Beginning new tablet copy session on tablet f1f416f090284b8d92d53662cc7a3b97 from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 at {username='slave'} at 127.20.207.1:49483: session id = 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-f1f416f090284b8d92d53662cc7a3b97
I20240502 08:18:28.740357 3227 tablet_replica.cc:330] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: stopping tablet replica
I20240502 08:18:28.740571 3227 raft_consensus.cc:2229] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:28.740796 3227 raft_consensus.cc:2258] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:28.741207 30310 tablet_copy_source_session.cc:205] T f1f416f090284b8d92d53662cc7a3b97 P 49c5c8f70e494edd9fcda40cc0af6cc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:28.750849 3218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f416f090284b8d92d53662cc7a3b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:28.754458 3227 ts_tablet_manager.cc:1875] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:28.758535 2821 raft_consensus.cc:1050] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4: attempting to promote NON_VOTER b01e7c8a302d41078fa39cfef665ff59 to VOTER
I20240502 08:18:28.759271 2821 consensus_queue.cc:237] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:28.784016 3218 tablet_copy_client.cc:783] T f1f416f090284b8d92d53662cc7a3b97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:28.784760 3218 tablet_copy_client.cc:647] T f1f416f090284b8d92d53662cc7a3b97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:28.786247 3217 tablet_bootstrap.cc:492] T a52fe5caeb364b868a1965633a7da09e P b01e7c8a302d41078fa39cfef665ff59: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:28.787379 3217 tablet_bootstrap.cc:492] T a52fe5caeb364b868a1965633a7da09e P b01e7c8a302d41078fa39cfef665ff59: Bootstrap complete.
I20240502 08:18:28.792517 3218 tablet_copy_client.cc:523] T f1f416f090284b8d92d53662cc7a3b97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:28.796322 3218 tablet_bootstrap.cc:492] T f1f416f090284b8d92d53662cc7a3b97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap starting.
W20240502 08:18:28.801208 27194 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.207.4:33309, user_credentials={real_user=slave}} blocked reactor thread for 64581.2us
I20240502 08:18:28.801728 3217 ts_tablet_manager.cc:1367] T a52fe5caeb364b868a1965633a7da09e P b01e7c8a302d41078fa39cfef665ff59: Time spent bootstrapping tablet: real 0.080s user 0.008s sys 0.000s
I20240502 08:18:28.802574 27516 raft_consensus.cc:1261] T ecc002b16f4642159e199298acd38392 P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:28.803126 2589 consensus_queue.cc:1035] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:28.803452 3217 raft_consensus.cc:348] T a52fe5caeb364b868a1965633a7da09e P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:28.803692 3217 raft_consensus.cc:729] T a52fe5caeb364b868a1965633a7da09e P b01e7c8a302d41078fa39cfef665ff59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b01e7c8a302d41078fa39cfef665ff59, State: Initialized, Role: LEARNER
I20240502 08:18:28.808312 3227 ts_tablet_manager.cc:1888] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:28.803887 3217 consensus_queue.cc:260] T a52fe5caeb364b868a1965633a7da09e P b01e7c8a302d41078fa39cfef665ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { replace: true } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: NON_VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: true } }
I20240502 08:18:28.810204 30156 raft_consensus.cc:1261] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:28.810704 2589 consensus_queue.cc:1035] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:28.811343 30025 raft_consensus.cc:1261] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:28.818672 2589 consensus_queue.cc:1035] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:28.830078 3245 ts_tablet_manager.cc:902] T cef42b07900c4b4e8589f284c011c300 P b01e7c8a302d41078fa39cfef665ff59: Initiating tablet copy from peer 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
F20240502 08:18:28.830644 3245 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0a5a0175799e478c9f53ecc82ba090ae.queue_time_us, tablet-download-pool-0a5a0175799e478c9f53ecc82ba090ae.queue_time_us) (tablet-download-pool-0a5a0175799e478c9f53ecc82ba090ae.run_cpu_time_us, tablet-download-pool-0a5a0175799e478c9f53ecc82ba090ae.run_cpu_time_us) (tablet-download-pool-0a5a0175799e478c9f53ecc82ba090ae.run_wall_time_us, tablet-download-pool-0a5a0175799e478c9f53ecc82ba090ae.run_wall_time_us) (tablet-download-pool-0d33916984034ab99f957026b3e30d38.queue_time_us, tablet-download-pool-0d33916984034ab99f957026b3e30d38.queue_time_us) (tablet-download-pool-0d33916984034ab99f957026b3e30d38.run_cpu_time_us, tablet-download-pool-0d33916984034ab99f957026b3e30d38.run_cpu_time_us) (tablet-download-pool-0d33916984034ab99f957026b3e30d38.run_wall_time_us, tablet-download-pool-0d33916984034ab99f957026b3e30d38.run_wall_time_us) (tablet-download-pool-16b897307b4c4cbea72f814f64f1e7d7.queue_time_us, tablet-download-pool-16b897307b4c4cbea72f814f64f1e7d7.queue_time_us) (tablet-download-pool-16b897307b4c4cbea72f814f64f1e7d7.run_cpu_time_us, tablet-download-pool-16b897307b4c4cbea72f814f64f1e7d7.run_cpu_time_us) (tablet-download-pool-16b897307b4c4cbea72f814f64f1e7d7.run_wall_time_us, tablet-download-pool-16b897307b4c4cbea72f814f64f1e7d7.run_wall_time_us) (tablet-download-pool-4c78e068ec3a451c892d58174b3f837c.queue_time_us, tablet-download-pool-4c78e068ec3a451c892d58174b3f837c.queue_time_us) (tablet-download-pool-4c78e068ec3a451c892d58174b3f837c.run_cpu_time_us, tablet-download-pool-4c78e068ec3a451c892d58174b3f837c.run_cpu_time_us) (tablet-download-pool-4c78e068ec3a451c892d58174b3f837c.run_wall_time_us, tablet-download-pool-4c78e068ec3a451c892d58174b3f837c.run_wall_time_us) (tablet-download-pool-8d309785afd8467dab3418bb63c95030.queue_time_us, tablet-download-pool-8d309785afd8467dab3418bb63c95030.queue_time_us) (tablet-download-pool-8d309785afd8467dab3418bb63c95030.run_cpu_time_us, tablet-download-pool-8d309785afd8467dab3418bb63c95030.run_cpu_time_us) (tablet-download-pool-8d309785afd8467dab3418bb63c95030.run_wall_time_us, tablet-download-pool-8d309785afd8467dab3418bb63c95030.run_wall_time_us) (tablet-download-pool-9bd5657c5cfd4ca4a78cdfdc503b91a8.queue_time_us, tablet-download-pool-9bd5657c5cfd4ca4a78cdfdc503b91a8.queue_time_us) (tablet-download-pool-9bd5657c5cfd4ca4a78cdfdc503b91a8.run_cpu_time_us, tablet-download-pool-9bd5657c5cfd4ca4a78cdfdc503b91a8.run_cpu_time_us) (tablet-download-pool-9bd5657c5cfd4ca4a78cdfdc503b91a8.run_wall_time_us, tablet-download-pool-9bd5657c5cfd4ca4a78cdfdc503b91a8.run_wall_time_us) (tablet-download-pool-a52fe5caeb364b868a1965633a7da09e.queue_time_us, tablet-download-pool-a52fe5caeb364b868a1965633a7da09e.queue_time_us) (tablet-download-pool-a52fe5caeb364b868a1965633a7da09e.run_cpu_time_us, tablet-download-pool-a52fe5caeb364b868a1965633a7da09e.run_cpu_time_us) (tablet-download-pool-a52fe5caeb364b868a1965633a7da09e.run_wall_time_us, tablet-download-pool-a52fe5caeb364b868a1965633a7da09e.run_wall_time_us) (tablet-download-pool-ca8d1416a54344feae081ed7b19dd1de.queue_time_us, tablet-download-pool-ca8d1416a54344feae081ed7b19dd1de.queue_time_us) (tablet-download-pool-ca8d1416a54344feae081ed7b19dd1de.run_cpu_time_us, tablet-download-pool-ca8d1416a54344feae081ed7b19dd1de.run_cpu_time_us) (tablet-download-pool-ca8d1416a54344feae081ed7b19dd1de.run_wall_time_us, tablet-download-pool-ca8d1416a54344feae081ed7b19dd1de.run_wall_time_us) (tablet-download-pool-cef42b07900c4b4e8589f284c011c300.queue_time_us, tablet-download-pool-cef42b07900c4b4e8589f284c011c300.queue_time_us) (tablet-download-pool-d00f25f29f5d4db2a2261dd37c0793fa.queue_time_us, tablet-download-pool-d00f25f29f5d4db2a2261dd37c0793fa.queue_time_us) (tablet-download-pool-d00f25f29f5d4db2a2261dd37c0793fa.run_cpu_time_us, tablet-download-pool-d00f25f29f5d4db2a2261dd37c0793fa.run_cpu_time_us) (tablet-download-pool-d00f25f29f5d4db2a2261dd37c0793fa.run_wall_time_us, tablet-download-pool-d00f25f29f5d4db2a2261dd37c0793fa.run_wall_time_us) (tablet-download-pool-e67c99e57a73443da13527ed33326a4e.queue_time_us, tablet-download-pool-e67c99e57a73443da13527ed33326a4e.queue_time_us) (tablet-download-pool-e67c99e57a73443da13527ed33326a4e.run_cpu_time_us, tablet-download-pool-e67c99e57a73443da13527ed33326a4e.run_cpu_time_us) (tablet-download-pool-e67c99e57a73443da13527ed33326a4e.run_wall_time_us, tablet-download-pool-e67c99e57a73443da13527ed33326a4e.run_wall_time_us) (tablet-download-pool-e7064865693149b980ba54fb8e65bc48.queue_time_us, tablet-download-pool-e7064865693149b980ba54fb8e65bc48.queue_time_us) (tablet-download-pool-e7064865693149b980ba54fb8e65bc48.run_cpu_time_us, tablet-download-pool-e7064865693149b980ba54fb8e65bc48.run_cpu_time_us) (tablet-download-pool-e7064865693149b980ba54fb8e65bc48.run_wall_time_us, tablet-download-pool-e7064865693149b980ba54fb8e65bc48.run_wall_time_us) (tablet-download-pool-ecc002b16f4642159e199298acd38392.queue_time_us, tablet-download-pool-ecc002b16f4642159e199298acd38392.queue_time_us) (tablet-download-pool-ecc002b16f4642159e199298acd38392.run_cpu_time_us, tablet-download-pool-ecc002b16f4642159e199298acd38392.run_cpu_time_us) (tablet-download-pool-ecc002b16f4642159e199298acd38392.run_wall_time_us, tablet-download-pool-ecc002b16f4642159e199298acd38392.run_wall_time_us) (tablet-download-pool-f63eec7e63fe4e71b1958903e0d2cb00.queue_time_us, tablet-download-pool-f63eec7e63fe4e71b1958903e0d2cb00.queue_time_us) (tablet-download-pool-f63eec7e63fe4e71b1958903e0d2cb00.run_cpu_time_us, tablet-download-pool-f63eec7e63fe4e71b1958903e0d2cb00.run_cpu_time_us) (tablet-download-pool-f63eec7e63fe4e71b1958903e0d2cb00.run_wall_time_us, tablet-download-pool-f63eec7e63fe4e71b1958903e0d2cb00.run_wall_time_us) (tablet-download-pool-feb9cd278950480fb63be0be950d190b.queue_time_us, tablet-download-pool-feb9cd278950480fb63be0be950d190b.queue_time_us) (tablet-download-pool-feb9cd278950480fb63be0be950d190b.run_cpu_time_us, tablet-download-pool-feb9cd278950480fb63be0be950d190b.run_cpu_time_us) (tablet-download-pool-feb9cd278950480fb63be0be950d190b.run_wall_time_us, tablet-download-pool-feb9cd278950480fb63be0be950d190b.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 08:18:28.833961 30025 consensus_queue.cc:237] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
@ 0x7f50ee193dcd google::LogMessage::Fail() at ??:0
I20240502 08:18:28.837107 30157 raft_consensus.cc:1261] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
@ 0x7f50ee197b93 google::LogMessage::SendToLog() at ??:0
I20240502 08:18:28.839051 2281 consensus_queue.cc:1035] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
@ 0x7f50ee1937cc google::LogMessage::Flush() at ??:0
@ 0x7f50ee194f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 08:18:28.842062 2281 raft_consensus.cc:2932] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:28.845875 30157 raft_consensus.cc:2932] T f1f5650965e841ad98d857ca32299037 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
@ 0x7f50f0ed4742 kudu::TraceMetrics::InternName() at ??:0
I20240502 08:18:28.851481 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f1f5650965e841ad98d857ca32299037 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:28.856463 27112 catalog_manager.cc:5484] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 3 to 5, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New cstate: current_term: 2 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
@ 0x7f50f0ea0f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 08:18:28.862937 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet f1f5650965e841ad98d857ca32299037 with delete_type TABLET_DATA_TOMBSTONED (TS 49c5c8f70e494edd9fcda40cc0af6cc4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52730
I20240502 08:18:28.836499 30422 raft_consensus.cc:1261] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:18:28.870862 2281 consensus_queue.cc:1035] T f1f5650965e841ad98d857ca32299037 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:28.872000 30422 raft_consensus.cc:2932] T f1f5650965e841ad98d857ca32299037 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } }
I20240502 08:18:28.877841 3248 tablet_replica.cc:330] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4: stopping tablet replica
I20240502 08:18:28.878093 3248 raft_consensus.cc:2229] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:28.878254 3248 raft_consensus.cc:2258] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:28.879155 3248 ts_tablet_manager.cc:1875] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
@ 0x7f50f0e9acd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 08:18:28.884532 3217 ts_tablet_manager.cc:1398] T a52fe5caeb364b868a1965633a7da09e P b01e7c8a302d41078fa39cfef665ff59: Time spent starting tablet: real 0.083s user 0.002s sys 0.000s
I20240502 08:18:28.884729 3227 log.cc:1196] T 51201cc579394a2b819278a5943485a0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-7/wal/wals/51201cc579394a2b819278a5943485a0
I20240502 08:18:28.885813 27100 catalog_manager.cc:4830] TS 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619): tablet 51201cc579394a2b819278a5943485a0 (table rebalancer_extra_table_52 [id=04082d4941a343ab8e88027c2100774d]) successfully deleted
I20240502 08:18:28.887059 3248 ts_tablet_manager.cc:1888] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 08:18:28.887184 3248 log.cc:1196] T f1f5650965e841ad98d857ca32299037 P 49c5c8f70e494edd9fcda40cc0af6cc4: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-6/wal/wals/f1f5650965e841ad98d857ca32299037
I20240502 08:18:28.888082 27100 catalog_manager.cc:4830] TS 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): tablet f1f5650965e841ad98d857ca32299037 (table rebalancer_extra_table_20 [id=aff097c8fa414c8aa9c675914ae4280a]) successfully deleted
I20240502 08:18:28.888792 30179 tablet_copy_service.cc:342] P 84cda20738d2452da166e31fc6da57a6: Request end of tablet copy session b01e7c8a302d41078fa39cfef665ff59-a52fe5caeb364b868a1965633a7da09e received from {username='slave'} at 127.20.207.3:35877
I20240502 08:18:28.888912 30179 tablet_copy_service.cc:433] P 84cda20738d2452da166e31fc6da57a6: ending tablet copy session b01e7c8a302d41078fa39cfef665ff59-a52fe5caeb364b868a1965633a7da09e on tablet a52fe5caeb364b868a1965633a7da09e with peer b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:28.891975 2589 raft_consensus.cc:2932] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
@ 0x7f51002ed8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 08:18:28.895234 30026 raft_consensus.cc:2932] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
@ 0x7f5100305daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f5100430480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 08:18:28.907124 27112 catalog_manager.cc:5484] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
@ 0x7f5100468a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 08:18:28.911280 3218 tablet_bootstrap.cc:492] T f1f416f090284b8d92d53662cc7a3b97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:28.911723 3218 tablet_bootstrap.cc:492] T f1f416f090284b8d92d53662cc7a3b97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Bootstrap complete.
I20240502 08:18:28.912009 3218 ts_tablet_manager.cc:1367] T f1f416f090284b8d92d53662cc7a3b97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent bootstrapping tablet: real 0.116s user 0.006s sys 0.010s
I20240502 08:18:28.912676 3218 raft_consensus.cc:348] T f1f416f090284b8d92d53662cc7a3b97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:28.912955 3218 raft_consensus.cc:729] T f1f416f090284b8d92d53662cc7a3b97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Initialized, Role: LEARNER
I20240502 08:18:28.913149 3218 consensus_queue.cc:260] T f1f416f090284b8d92d53662cc7a3b97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
@ 0x7f510044b981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f510044b70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f50f0ea4b75 kudu::ThreadPool::DispatchThread() at ??:0
I20240502 08:18:28.925244 30156 raft_consensus.cc:2932] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
@ 0x7f50f0ea819c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f50f0ea8019 std::_Function_handler<>::_M_invoke() at ??:0
I20240502 08:18:28.933848 3218 ts_tablet_manager.cc:1398] T f1f416f090284b8d92d53662cc7a3b97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Time spent starting tablet: real 0.022s user 0.002s sys 0.001s
I20240502 08:18:28.933423 27520 raft_consensus.cc:2932] T ecc002b16f4642159e199298acd38392 P b01e7c8a302d41078fa39cfef665ff59 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { replace: true } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 08:18:28.936946 30310 tablet_copy_service.cc:342] P 49c5c8f70e494edd9fcda40cc0af6cc4: Request end of tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-f1f416f090284b8d92d53662cc7a3b97 received from {username='slave'} at 127.20.207.1:49483
I20240502 08:18:28.937062 30310 tablet_copy_service.cc:433] P 49c5c8f70e494edd9fcda40cc0af6cc4: ending tablet copy session 1c0e4a6cec3c4ebb92a5abf1d8b8ba60-f1f416f090284b8d92d53662cc7a3b97 on tablet f1f416f090284b8d92d53662cc7a3b97 with peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60
I20240502 08:18:28.937419 3194 raft_consensus.cc:1050] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1: attempting to promote NON_VOTER 84cda20738d2452da166e31fc6da57a6 to VOTER
I20240502 08:18:28.938566 3194 consensus_queue.cc:237] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
@ 0x7f50f0e8301e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f50f42086db start_thread at ??:0
@ 0x7f50ed04671f clone at ??:0
I20240502 08:18:28.950543 30026 raft_consensus.cc:1261] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:28.951668 3195 consensus_queue.cc:1035] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:28.956272 30156 raft_consensus.cc:1261] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6 [term 1 LEARNER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:28.959426 3195 consensus_queue.cc:1035] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:28.960124 30290 consensus_queue.cc:237] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:28.976195 30026 raft_consensus.cc:1261] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 49c5c8f70e494edd9fcda40cc0af6cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:28.977993 2948 consensus_queue.cc:1035] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.987085 2948 raft_consensus.cc:2932] T ecc002b16f4642159e199298acd38392 P 49c5c8f70e494edd9fcda40cc0af6cc4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:28.990840 30026 raft_consensus.cc:2932] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:29.000659 27114 catalog_manager.cc:5484] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 3 to 4, VOTER 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) evicted. New cstate: current_term: 1 leader_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } } }
I20240502 08:18:29.003046 27100 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ecc002b16f4642159e199298acd38392 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:29.022506 30139 tablet_service.cc:1514] Processing DeleteTablet for tablet ecc002b16f4642159e199298acd38392 with delete_type TABLET_DATA_TOMBSTONED (TS 84cda20738d2452da166e31fc6da57a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48808
I20240502 08:18:29.032001 3268 tablet_replica.cc:330] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6: stopping tablet replica
I20240502 08:18:29.031635 3195 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:29.032205 3268 raft_consensus.cc:2229] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:29.032449 3268 raft_consensus.cc:2258] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:29.036118 3268 ts_tablet_manager.cc:1875] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:29.036945 30026 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:29.045886 27112 catalog_manager.cc:5484] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:29.047458 27255 raft_consensus.cc:1203] T f1f416f090284b8d92d53662cc7a3b97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:18:29.053201 3268 ts_tablet_manager.cc:1888] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 08:18:29.053313 3268 log.cc:1196] T ecc002b16f4642159e199298acd38392 P 84cda20738d2452da166e31fc6da57a6: Deleting WAL directory at /tmp/dist-test-taskZkl2nR/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714637792427645-21308-0/raft_consensus-itest-cluster/ts-5/wal/wals/ecc002b16f4642159e199298acd38392
I20240502 08:18:29.059419 27387 consensus_queue.cc:237] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:29.062642 30156 raft_consensus.cc:1261] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:29.063455 3196 consensus_queue.cc:1035] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:29.064579 30156 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 84cda20738d2452da166e31fc6da57a6 (127.20.207.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:29.066304 30026 raft_consensus.cc:1261] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 83e171a5e16c4008923f1d43fc5ce7b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:29.070639 27103 catalog_manager.cc:4830] TS 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899): tablet ecc002b16f4642159e199298acd38392 (table rebalancer_extra_table_34 [id=5ebae50d574942c6976b1ce3e50307d5]) successfully deleted
I20240502 08:18:29.070402 3195 consensus_queue.cc:1035] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:29.089084 3196 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:29.093904 27102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6cf739b314d84db08bc4cb32b0e5b0f4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:29.096791 3271 ts_tablet_manager.cc:902] T c30720c1b245438db3eb4bea2b8bd930 P 49c5c8f70e494edd9fcda40cc0af6cc4: Initiating tablet copy from peer 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:29.096463 30028 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
F20240502 08:18:29.097107 3271 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-09aa1b5d99114b3095be08c9547fbb54.queue_time_us, tablet-download-pool-09aa1b5d99114b3095be08c9547fbb54.queue_time_us) (tablet-download-pool-09aa1b5d99114b3095be08c9547fbb54.run_cpu_time_us, tablet-download-pool-09aa1b5d99114b3095be08c9547fbb54.run_cpu_time_us) (tablet-download-pool-09aa1b5d99114b3095be08c9547fbb54.run_wall_time_us, tablet-download-pool-09aa1b5d99114b3095be08c9547fbb54.run_wall_time_us) (tablet-download-pool-15a333c65455473fbdc26d1ec137aebc.queue_time_us, tablet-download-pool-15a333c65455473fbdc26d1ec137aebc.queue_time_us) (tablet-download-pool-15a333c65455473fbdc26d1ec137aebc.run_cpu_time_us, tablet-download-pool-15a333c65455473fbdc26d1ec137aebc.run_cpu_time_us) (tablet-download-pool-15a333c65455473fbdc26d1ec137aebc.run_wall_time_us, tablet-download-pool-15a333c65455473fbdc26d1ec137aebc.run_wall_time_us) (tablet-download-pool-2dfa509d47e54b0f9ccacc9954981fb6.queue_time_us, tablet-download-pool-2dfa509d47e54b0f9ccacc9954981fb6.queue_time_us) (tablet-download-pool-2dfa509d47e54b0f9ccacc9954981fb6.run_cpu_time_us, tablet-download-pool-2dfa509d47e54b0f9ccacc9954981fb6.run_cpu_time_us) (tablet-download-pool-2dfa509d47e54b0f9ccacc9954981fb6.run_wall_time_us, tablet-download-pool-2dfa509d47e54b0f9ccacc9954981fb6.run_wall_time_us) (tablet-download-pool-3047a6db2c9d438eb5720d848b9ea5bd.queue_time_us, tablet-download-pool-3047a6db2c9d438eb5720d848b9ea5bd.queue_time_us) (tablet-download-pool-3047a6db2c9d438eb5720d848b9ea5bd.run_cpu_time_us, tablet-download-pool-3047a6db2c9d438eb5720d848b9ea5bd.run_cpu_time_us) (tablet-download-pool-3047a6db2c9d438eb5720d848b9ea5bd.run_wall_time_us, tablet-download-pool-3047a6db2c9d438eb5720d848b9ea5bd.run_wall_time_us) (tablet-download-pool-7c233aeb1e58418ba7b01f48917fabad.queue_time_us, tablet-download-pool-7c233aeb1e58418ba7b01f48917fabad.queue_time_us) (tablet-download-pool-7c233aeb1e58418ba7b01f48917fabad.run_cpu_time_us, tablet-download-pool-7c233aeb1e58418ba7b01f48917fabad.run_cpu_time_us) (tablet-download-pool-7c233aeb1e58418ba7b01f48917fabad.run_wall_time_us, tablet-download-pool-7c233aeb1e58418ba7b01f48917fabad.run_wall_time_us) (tablet-download-pool-903502e57ab2430383807b0d46a39847.queue_time_us, tablet-download-pool-903502e57ab2430383807b0d46a39847.queue_time_us) (tablet-download-pool-903502e57ab2430383807b0d46a39847.run_cpu_time_us, tablet-download-pool-903502e57ab2430383807b0d46a39847.run_cpu_time_us) (tablet-download-pool-903502e57ab2430383807b0d46a39847.run_wall_time_us, tablet-download-pool-903502e57ab2430383807b0d46a39847.run_wall_time_us) (tablet-download-pool-a6dce110666a4a038f5f46d48a3ec7af.queue_time_us, tablet-download-pool-a6dce110666a4a038f5f46d48a3ec7af.queue_time_us) (tablet-download-pool-a6dce110666a4a038f5f46d48a3ec7af.run_cpu_time_us, tablet-download-pool-a6dce110666a4a038f5f46d48a3ec7af.run_cpu_time_us) (tablet-download-pool-a6dce110666a4a038f5f46d48a3ec7af.run_wall_time_us, tablet-download-pool-a6dce110666a4a038f5f46d48a3ec7af.run_wall_time_us) (tablet-download-pool-a793eaaeba89458baf8af7caf3f24c70.queue_time_us, tablet-download-pool-a793eaaeba89458baf8af7caf3f24c70.queue_time_us) (tablet-download-pool-a793eaaeba89458baf8af7caf3f24c70.run_cpu_time_us, tablet-download-pool-a793eaaeba89458baf8af7caf3f24c70.run_cpu_time_us) (tablet-download-pool-a793eaaeba89458baf8af7caf3f24c70.run_wall_time_us, tablet-download-pool-a793eaaeba89458baf8af7caf3f24c70.run_wall_time_us) (tablet-download-pool-b9d6207b6bb3436f996a069ff910a56e.queue_time_us, tablet-download-pool-b9d6207b6bb3436f996a069ff910a56e.queue_time_us) (tablet-download-pool-b9d6207b6bb3436f996a069ff910a56e.run_cpu_time_us, tablet-download-pool-b9d6207b6bb3436f996a069ff910a56e.run_cpu_time_us) (tablet-download-pool-b9d6207b6bb3436f996a069ff910a56e.run_wall_time_us, tablet-download-pool-b9d6207b6bb3436f996a069ff910a56e.run_wall_time_us) (tablet-download-pool-c2776f857a5f47e9b8022639dc54757d.queue_time_us, tablet-download-pool-c2776f857a5f47e9b8022639dc54757d.queue_time_us) (tablet-download-pool-c2776f857a5f47e9b8022639dc54757d.run_cpu_time_us, tablet-download-pool-c2776f857a5f47e9b8022639dc54757d.run_cpu_time_us) (tablet-download-pool-c2776f857a5f47e9b8022639dc54757d.run_wall_time_us, tablet-download-pool-c2776f857a5f47e9b8022639dc54757d.run_wall_time_us) (tablet-download-pool-c30720c1b245438db3eb4bea2b8bd930.queue_time_us, tablet-download-pool-c30720c1b245438db3eb4bea2b8bd930.queue_time_us) (tablet-download-pool-d3a3473a93f14baaac48b9b7f47ac2a1.queue_time_us, tablet-download-pool-d3a3473a93f14baaac48b9b7f47ac2a1.queue_time_us) (tablet-download-pool-d3a3473a93f14baaac48b9b7f47ac2a1.run_cpu_time_us, tablet-download-pool-d3a3473a93f14baaac48b9b7f47ac2a1.run_cpu_time_us) (tablet-download-pool-d3a3473a93f14baaac48b9b7f47ac2a1.run_wall_time_us, tablet-download-pool-d3a3473a93f14baaac48b9b7f47ac2a1.run_wall_time_us) (tablet-download-pool-e80b9a98bdd041b1ac97af055951a92a.queue_time_us, tablet-download-pool-e80b9a98bdd041b1ac97af055951a92a.queue_time_us) (tablet-download-pool-e80b9a98bdd041b1ac97af055951a92a.run_cpu_time_us, tablet-download-pool-e80b9a98bdd041b1ac97af055951a92a.run_cpu_time_us) (tablet-download-pool-e80b9a98bdd041b1ac97af055951a92a.run_wall_time_us, tablet-download-pool-e80b9a98bdd041b1ac97af055951a92a.run_wall_time_us) (tablet-download-pool-eca2618572fe4900bd420bb59c22381f.queue_time_us, tablet-download-pool-eca2618572fe4900bd420bb59c22381f.queue_time_us) (tablet-download-pool-eca2618572fe4900bd420bb59c22381f.run_cpu_time_us, tablet-download-pool-eca2618572fe4900bd420bb59c22381f.run_cpu_time_us) (tablet-download-pool-eca2618572fe4900bd420bb59c22381f.run_wall_time_us, tablet-download-pool-eca2618572fe4900bd420bb59c22381f.run_wall_time_us) (tablet-download-pool-fef4f56761bb458c8d84807feb38a617.queue_time_us, tablet-download-pool-fef4f56761bb458c8d84807feb38a617.queue_time_us) (tablet-download-pool-fef4f56761bb458c8d84807feb38a617.run_cpu_time_us, tablet-download-pool-fef4f56761bb458c8d84807feb38a617.run_cpu_time_us) (tablet-download-pool-fef4f56761bb458c8d84807feb38a617.run_wall_time_us, tablet-download-pool-fef4f56761bb458c8d84807feb38a617.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240502 08:18:29.097892 30159 raft_consensus.cc:2932] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } }
I20240502 08:18:29.100379 27115 catalog_manager.cc:5484] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: config changed from index 8 to 9, VOTER b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) evicted. New cstate: current_term: 1 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
*** Check failure stack trace: ***
@ 0x7fe0c7b11dcd google::LogMessage::Fail() at ??:0
@ 0x7fe0c7b15b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fe0c7b117cc google::LogMessage::Flush() at ??:0
@ 0x7fe0c7b12f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe0ca852742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fe0ca81ef8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fe0ca818cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fe0d9c6b8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fe0d9c83daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fe0d9dae480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fe0d9de6a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fe0d9dc9981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fe0d9dc970e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fe0ca822b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe0ca82619c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe0ca826019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fe0ca80101e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe0cdb866db start_thread at ??:0
@ 0x7fe0c69c471f clone at ??:0
W20240502 08:18:29.422485 30358 connection.cc:535] client connection to 127.20.207.3:44469 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.422726 30358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.425877 29964 connection.cc:535] client connection to 127.20.207.3:44469 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.426192 29964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.427649 30096 connection.cc:535] client connection to 127.20.207.3:44469 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.427855 30096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.434454 27326 connection.cc:535] client connection to 127.20.207.3:44469 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.434695 27326 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.435741 27587 connection.cc:535] client connection to 127.20.207.3:44469 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.435938 27587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.443799 27194 connection.cc:535] client connection to 127.20.207.3:44469 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.444001 27194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.445974 27103 connection.cc:535] client connection to 127.20.207.3:44469 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.446230 27103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.446981 3318 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:18:29.440141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.207.3:44469 (local address 127.20.207.6:52037)
0502 08:18:29.440504 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0502 08:18:29.440518 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0502 08:18:29.440604 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:18:29.446109 (+ 5505us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":864,"threads_started":1}
W20240502 08:18:29.447016 3310 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:18:29.425213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.207.3:44469 (local address 127.20.207.8:34071)
0502 08:18:29.428218 (+ 3005us) negotiation.cc:107] Waiting for socket to connect
0502 08:18:29.428243 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0502 08:18:29.428436 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:18:29.446109 (+ 17673us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":213,"threads_started":1}
W20240502 08:18:29.447150 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.447261 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.447450 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.447616 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.447693 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.450398 27103 catalog_manager.cc:4573] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6cf739b314d84db08bc4cb32b0e5b0f4: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:29.456869 3312 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:18:29.427949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.207.3:44469 (local address 127.20.207.5:57331)
0502 08:18:29.438967 (+ 11018us) negotiation.cc:107] Waiting for socket to connect
0502 08:18:29.438985 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0502 08:18:29.439115 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:18:29.449512 (+ 10397us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":10919,"thread_start_us":10825,"threads_started":1}
W20240502 08:18:29.457031 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.457132 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.457190 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.457670 3320 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:18:29.445001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.207.3:44469 (local address 127.20.207.1:40919)
0502 08:18:29.445381 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0502 08:18:29.445395 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0502 08:18:29.445504 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:18:29.457176 (+ 11672us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":225,"threads_started":1}
W20240502 08:18:29.457801 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.457904 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.458021 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.458099 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.458169 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.459013 3317 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:18:29.436579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.207.3:44469 (local address 127.20.207.2:54249)
0502 08:18:29.437022 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0502 08:18:29.437040 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0502 08:18:29.437158 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:18:29.447335 (+ 10177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":247,"threads_started":1}
W20240502 08:18:29.459270 27326 consensus_peers.cc:485] T 8d309785afd8467dab3418bb63c95030 P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.459386 27326 consensus_peers.cc:485] T 47b5771de7034e10acf2f40d0ac3c4d3 P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.459471 27326 consensus_peers.cc:485] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.459568 27326 consensus_peers.cc:485] T fee0386181d84b18803effc8af1bbbc4 P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.459645 27326 consensus_peers.cc:485] T 0ed6b8adc62f471696cbf0330524e353 P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.462412 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.465775 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.499562 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.512821 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.561345 29961 connection.cc:535] client connection to 127.20.207.7:40343 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.562542 27323 connection.cc:535] client connection to 127.20.207.7:40343 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.563040 27584 connection.cc:535] client connection to 127.20.207.7:40343 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.564289 30355 connection.cc:535] client connection to 127.20.207.7:40343 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.564299 30093 connection.cc:535] client connection to 127.20.207.7:40343 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.565598 27191 connection.cc:535] client connection to 127.20.207.7:40343 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.566346 3312 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:18:29.562449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.207.7:40343 (local address 127.20.207.5:45949)
0502 08:18:29.562744 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0502 08:18:29.562758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0502 08:18:29.562872 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:18:29.566200 (+ 3328us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":226}
W20240502 08:18:29.566670 3328 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:18:29.563889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.207.7:40343 (local address 127.20.207.4:57171)
0502 08:18:29.564051 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0502 08:18:29.564067 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0502 08:18:29.564164 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:18:29.565750 (+ 1586us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":50}
W20240502 08:18:29.566776 3317 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:18:29.563550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.207.7:40343 (local address 127.20.207.2:46781)
0502 08:18:29.563701 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0502 08:18:29.563717 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0502 08:18:29.563835 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:18:29.566607 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":31}
W20240502 08:18:29.566977 27323 consensus_peers.cc:485] T f63eec7e63fe4e71b1958903e0d2cb00 P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.567083 27323 consensus_peers.cc:485] T 119ad5932c314235b85177cceeb2277c P 83e171a5e16c4008923f1d43fc5ce7b1 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.567322 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.567420 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.567905 3318 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:18:29.565096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.207.7:40343 (local address 127.20.207.6:33967)
0502 08:18:29.565322 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0502 08:18:29.565337 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0502 08:18:29.565431 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:18:29.567714 (+ 2283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":128}
W20240502 08:18:29.568078 3310 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:18:29.565064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.207.7:40343 (local address 127.20.207.8:54015)
0502 08:18:29.565362 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0502 08:18:29.565375 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0502 08:18:29.565454 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:18:29.567917 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":200}
W20240502 08:18:29.568069 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:29.568181 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.568284 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.568313 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.568379 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.568395 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.568445 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.568509 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.568493 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.568696 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.568814 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.568895 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.568975 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.569133 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.569234 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.569316 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.569375 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.569442 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.577309 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.599632 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:29.610209 27255 consensus_queue.cc:237] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } }
W20240502 08:18:29.612733 27193 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bfa383343184f18969fa52ed97ab07c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:29.612923 30025 raft_consensus.cc:1261] T 3bfa383343184f18969fa52ed97ab07c P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:29.613839 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:29.614979 3238 consensus_queue.cc:1035] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:29.615216 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:29.617009 3229 raft_consensus.cc:2932] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:29.617565 30025 raft_consensus.cc:2932] T 3bfa383343184f18969fa52ed97ab07c P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } } }
I20240502 08:18:29.622536 27110 catalog_manager.cc:5484] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: config changed from index -1 to 2, NON_VOTER 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) added. New cstate: current_term: 1 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: NON_VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:29.675925 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.696049 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.734361 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:29.745962 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:29.909597 3434 raft_consensus.cc:482] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:29.909765 3434 raft_consensus.cc:504] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:29.910444 3434 leader_election.cc:288] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:29.911008 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0313e85d444e098bfa35be89a3ad97" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
I20240502 08:18:29.915544 2954 raft_consensus.cc:482] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:29.915792 2954 raft_consensus.cc:504] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:29.916893 2954 leader_election.cc:288] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:29.919878 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefcc470084f4ec9be6ba45d3ba22ce5" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
W20240502 08:18:29.921702 27194 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:29.921969 27194 leader_election.cc:302] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:29.922394 3229 raft_consensus.cc:2726] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:29.923590 30096 leader_election.cc:334] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:29.923749 30096 leader_election.cc:302] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442
I20240502 08:18:29.924041 3164 raft_consensus.cc:2726] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:18:29.931895 3434 raft_consensus.cc:482] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:29.932006 3434 raft_consensus.cc:504] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:29.932513 3434 leader_election.cc:288] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:29.933349 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4b822c4aca4f7f92b73cc9279ddec5" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
W20240502 08:18:29.933889 27194 leader_election.cc:334] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:29.934103 27194 leader_election.cc:302] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442
I20240502 08:18:29.934347 3434 raft_consensus.cc:2726] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:29.941991 3087 raft_consensus.cc:482] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:29.942221 3087 raft_consensus.cc:504] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:29.942968 3087 leader_election.cc:288] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:29.943782 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0313e85d444e098bfa35be89a3ad97" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:29.943969 27257 raft_consensus.cc:2454] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 1.
I20240502 08:18:29.944344 27325 leader_election.cc:302] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
W20240502 08:18:29.944569 27326 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:29.944700 3087 raft_consensus.cc:2781] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:18:29.944802 3087 raft_consensus.cc:482] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:29.944885 3087 raft_consensus.cc:3037] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:29.947880 3087 raft_consensus.cc:504] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:29.950838 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0313e85d444e098bfa35be89a3ad97" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:29.951012 27257 raft_consensus.cc:3037] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:29.951139 3087 leader_election.cc:288] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 2 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:29.953650 27257 raft_consensus.cc:2454] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e171a5e16c4008923f1d43fc5ce7b1 in term 2.
I20240502 08:18:29.954056 27325 leader_election.cc:302] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 83e171a5e16c4008923f1d43fc5ce7b1; no voters:
W20240502 08:18:29.956327 27326 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:29.957212 3087 raft_consensus.cc:2781] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:29.957410 3087 raft_consensus.cc:686] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 2 LEADER]: Becoming Leader. State: Replica: 83e171a5e16c4008923f1d43fc5ce7b1, State: Running, Role: LEADER
I20240502 08:18:29.957679 3087 consensus_queue.cc:237] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:29.976315 3266 raft_consensus.cc:482] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:29.976470 3266 raft_consensus.cc:504] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:29.977128 3266 leader_election.cc:288] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:29.978683 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5345b931504cfdbb272189b719efa4" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:29.978480 27115 catalog_manager.cc:5484] T 5e0313e85d444e098bfa35be89a3ad97 P 83e171a5e16c4008923f1d43fc5ce7b1 reported cstate change: term changed from 1 to 2, leader changed from b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) to 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2). New cstate: current_term: 2 leader_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:29.979393 30358 leader_election.cc:334] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:29.979521 30358 leader_election.cc:302] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 84cda20738d2452da166e31fc6da57a6, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:29.979939 3266 raft_consensus.cc:2726] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:29.990135 3433 raft_consensus.cc:482] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:29.990348 3433 raft_consensus.cc:504] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:29.991210 3433 leader_election.cc:288] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:29.992651 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b9488fe8ac46099b7e8adf4d0b9cd5" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
W20240502 08:18:29.993256 27587 leader_election.cc:334] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:29.993510 27584 leader_election.cc:302] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 1f0bf4bc3c5146539146b825b048bff3, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:29.993775 3433 raft_consensus.cc:2726] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:29.994307 3055 raft_consensus.cc:482] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:29.994470 3055 raft_consensus.cc:504] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:29.995080 3055 leader_election.cc:288] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:29.995648 27387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b08a6b38f945b1b99f1d8ad1fd0ae0" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" is_pre_election: true
W20240502 08:18:29.996775 29964 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:29.996874 29964 leader_election.cc:302] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:29.997084 3055 raft_consensus.cc:2726] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:30.037063 3433 raft_consensus.cc:482] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.037209 3433 raft_consensus.cc:504] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:30.037742 3433 leader_election.cc:288] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:30.039376 27587 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:30.039870 27584 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.040014 27584 leader_election.cc:302] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:30.040308 3433 raft_consensus.cc:2726] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:30.047614 3460 ts_tablet_manager.cc:902] T 3bfa383343184f18969fa52ed97ab07c P 83e171a5e16c4008923f1d43fc5ce7b1: Initiating tablet copy from peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
F20240502 08:18:30.048188 3460 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-201d79895c2741dd9d9de0cc9f605a91.queue_time_us, tablet-download-pool-201d79895c2741dd9d9de0cc9f605a91.queue_time_us) (tablet-download-pool-201d79895c2741dd9d9de0cc9f605a91.run_cpu_time_us, tablet-download-pool-201d79895c2741dd9d9de0cc9f605a91.run_cpu_time_us) (tablet-download-pool-201d79895c2741dd9d9de0cc9f605a91.run_wall_time_us, tablet-download-pool-201d79895c2741dd9d9de0cc9f605a91.run_wall_time_us) (tablet-download-pool-2066c7aa052342a1aca3eb849aa251fc.queue_time_us, tablet-download-pool-2066c7aa052342a1aca3eb849aa251fc.queue_time_us) (tablet-download-pool-2066c7aa052342a1aca3eb849aa251fc.run_cpu_time_us, tablet-download-pool-2066c7aa052342a1aca3eb849aa251fc.run_cpu_time_us) (tablet-download-pool-2066c7aa052342a1aca3eb849aa251fc.run_wall_time_us, tablet-download-pool-2066c7aa052342a1aca3eb849aa251fc.run_wall_time_us) (tablet-download-pool-26545a868bd6415c9c9db4869b2841c7.queue_time_us, tablet-download-pool-26545a868bd6415c9c9db4869b2841c7.queue_time_us) (tablet-download-pool-26545a868bd6415c9c9db4869b2841c7.run_cpu_time_us, tablet-download-pool-26545a868bd6415c9c9db4869b2841c7.run_cpu_time_us) (tablet-download-pool-26545a868bd6415c9c9db4869b2841c7.run_wall_time_us, tablet-download-pool-26545a868bd6415c9c9db4869b2841c7.run_wall_time_us) (tablet-download-pool-2e9b191ce9234c028b990efd74a59e2e.queue_time_us, tablet-download-pool-2e9b191ce9234c028b990efd74a59e2e.queue_time_us) (tablet-download-pool-2e9b191ce9234c028b990efd74a59e2e.run_cpu_time_us, tablet-download-pool-2e9b191ce9234c028b990efd74a59e2e.run_cpu_time_us) (tablet-download-pool-2e9b191ce9234c028b990efd74a59e2e.run_wall_time_us, tablet-download-pool-2e9b191ce9234c028b990efd74a59e2e.run_wall_time_us) (tablet-download-pool-387e9ef5899541b0be27a41dbcf4475c.queue_time_us, tablet-download-pool-387e9ef5899541b0be27a41dbcf4475c.queue_time_us) (tablet-download-pool-387e9ef5899541b0be27a41dbcf4475c.run_cpu_time_us, tablet-download-pool-387e9ef5899541b0be27a41dbcf4475c.run_cpu_time_us) (tablet-download-pool-387e9ef5899541b0be27a41dbcf4475c.run_wall_time_us, tablet-download-pool-387e9ef5899541b0be27a41dbcf4475c.run_wall_time_us) (tablet-download-pool-3bfa383343184f18969fa52ed97ab07c.queue_time_us, tablet-download-pool-3bfa383343184f18969fa52ed97ab07c.queue_time_us) (tablet-download-pool-51201cc579394a2b819278a5943485a0.queue_time_us, tablet-download-pool-51201cc579394a2b819278a5943485a0.queue_time_us) (tablet-download-pool-51201cc579394a2b819278a5943485a0.run_cpu_time_us, tablet-download-pool-51201cc579394a2b819278a5943485a0.run_cpu_time_us) (tablet-download-pool-51201cc579394a2b819278a5943485a0.run_wall_time_us, tablet-download-pool-51201cc579394a2b819278a5943485a0.run_wall_time_us) (tablet-download-pool-57505255575249e698bf35929619423d.queue_time_us, tablet-download-pool-57505255575249e698bf35929619423d.queue_time_us) (tablet-download-pool-57505255575249e698bf35929619423d.run_cpu_time_us, tablet-download-pool-57505255575249e698bf35929619423d.run_cpu_time_us) (tablet-download-pool-57505255575249e698bf35929619423d.run_wall_time_us, tablet-download-pool-57505255575249e698bf35929619423d.run_wall_time_us) (tablet-download-pool-70f4c40bc22947468dfd15804a4e66cd.queue_time_us, tablet-download-pool-70f4c40bc22947468dfd15804a4e66cd.queue_time_us) (tablet-download-pool-70f4c40bc22947468dfd15804a4e66cd.run_cpu_time_us, tablet-download-pool-70f4c40bc22947468dfd15804a4e66cd.run_cpu_time_us) (tablet-download-pool-70f4c40bc22947468dfd15804a4e66cd.run_wall_time_us, tablet-download-pool-70f4c40bc22947468dfd15804a4e66cd.run_wall_time_us) (tablet-download-pool-81e73229c0d145e7bb81761bf4b820dc.queue_time_us, tablet-download-pool-81e73229c0d145e7bb81761bf4b820dc.queue_time_us) (tablet-download-pool-81e73229c0d145e7bb81761bf4b820dc.run_cpu_time_us, tablet-download-pool-81e73229c0d145e7bb81761bf4b820dc.run_cpu_time_us) (tablet-download-pool-81e73229c0d145e7bb81761bf4b820dc.run_wall_time_us, tablet-download-pool-81e73229c0d145e7bb81761bf4b820dc.run_wall_time_us) (tablet-download-pool-84953bd8daba4885be1f5f9ad4149aba.queue_time_us, tablet-download-pool-84953bd8daba4885be1f5f9ad4149aba.queue_time_us) (tablet-download-pool-84953bd8daba4885be1f5f9ad4149aba.run_cpu_time_us, tablet-download-pool-84953bd8daba4885be1f5f9ad4149aba.run_cpu_time_us) (tablet-download-pool-84953bd8daba4885be1f5f9ad4149aba.run_wall_time_us, tablet-download-pool-84953bd8daba4885be1f5f9ad4149aba.run_wall_time_us) (tablet-download-pool-9edac369e0ae4c3c990bbb8dd845b27c.queue_time_us, tablet-download-pool-9edac369e0ae4c3c990bbb8dd845b27c.queue_time_us) (tablet-download-pool-9edac369e0ae4c3c990bbb8dd845b27c.run_cpu_time_us, tablet-download-pool-9edac369e0ae4c3c990bbb8dd845b27c.run_cpu_time_us) (tablet-download-pool-9edac369e0ae4c3c990bbb8dd845b27c.run_wall_time_us, tablet-download-pool-9edac369e0ae4c3c990bbb8dd845b27c.run_wall_time_us) (tablet-download-pool-a9e5e67e3fe54491841b33d0d4bd047b.queue_time_us, tablet-download-pool-a9e5e67e3fe54491841b33d0d4bd047b.queue_time_us) (tablet-download-pool-a9e5e67e3fe54491841b33d0d4bd047b.run_cpu_time_us, tablet-download-pool-a9e5e67e3fe54491841b33d0d4bd047b.run_cpu_time_us) (tablet-download-pool-a9e5e67e3fe54491841b33d0d4bd047b.run_wall_time_us, tablet-download-pool-a9e5e67e3fe54491841b33d0d4bd047b.run_wall_time_us) (tablet-download-pool-dd644c6742854b79a212d858fa9df147.queue_time_us, tablet-download-pool-dd644c6742854b79a212d858fa9df147.queue_time_us) (tablet-download-pool-dd644c6742854b79a212d858fa9df147.run_cpu_time_us, tablet-download-pool-dd644c6742854b79a212d858fa9df147.run_cpu_time_us) (tablet-download-pool-dd644c6742854b79a212d858fa9df147.run_wall_time_us, tablet-download-pool-dd644c6742854b79a212d858fa9df147.run_wall_time_us) (tablet-download-pool-e5af418a0ae043a387b5030a00b47e91.queue_time_us, tablet-download-pool-e5af418a0ae043a387b5030a00b47e91.queue_time_us) (tablet-download-pool-e5af418a0ae043a387b5030a00b47e91.run_cpu_time_us, tablet-download-pool-e5af418a0ae043a387b5030a00b47e91.run_cpu_time_us) (tablet-download-pool-e5af418a0ae043a387b5030a00b47e91.run_wall_time_us, tablet-download-pool-e5af418a0ae043a387b5030a00b47e91.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f32c36a5dcd google::LogMessage::Fail() at ??:0
@ 0x7f32c36a9b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f32c36a57cc google::LogMessage::Flush() at ??:0
@ 0x7f32c36a6f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 08:18:30.058441 3055 raft_consensus.cc:482] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.058584 3055 raft_consensus.cc:504] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:30.059401 3055 leader_election.cc:288] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
@ 0x7f32c63e6742 kudu::TraceMetrics::InternName() at ??:0
I20240502 08:18:30.060364 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b9488fe8ac46099b7e8adf4d0b9cd5" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:30.060582 27650 raft_consensus.cc:2454] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:30.061076 29964 leader_election.cc:302] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, bf8ac71b15f740a0a928911360773442; no voters:
W20240502 08:18:30.061355 29964 leader_election.cc:334] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.061575 3055 raft_consensus.cc:2781] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:18:30.061659 3055 raft_consensus.cc:482] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.061760 3055 raft_consensus.cc:3037] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Advancing to term 2
@ 0x7f32c63b2f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 08:18:30.064841 3055 raft_consensus.cc:504] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:30.065397 3055 leader_election.cc:288] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:30.066131 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b9488fe8ac46099b7e8adf4d0b9cd5" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:30.066390 27650 raft_consensus.cc:3037] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:30.066789 29964 leader_election.cc:334] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
@ 0x7f32c63accd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 08:18:30.070459 27650 raft_consensus.cc:2454] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 2.
I20240502 08:18:30.070688 3055 raft_consensus.cc:482] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.070817 3055 raft_consensus.cc:504] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:30.070916 29964 leader_election.cc:302] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, bf8ac71b15f740a0a928911360773442; no voters: b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:30.071436 3055 leader_election.cc:288] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:30.071710 2943 raft_consensus.cc:2781] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:30.071842 2943 raft_consensus.cc:686] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:18:30.072093 2943 consensus_queue.cc:237] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:30.074072 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d122f68d36024614a017e8cc3a1224c1" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
W20240502 08:18:30.074995 29964 leader_election.cc:334] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.075096 29964 leader_election.cc:302] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 84cda20738d2452da166e31fc6da57a6, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:30.075356 3055 raft_consensus.cc:2726] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
@ 0x7f32d57ff8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 08:18:30.076784 2954 raft_consensus.cc:482] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.076927 2954 raft_consensus.cc:504] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:30.077804 2954 leader_election.cc:288] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:30.079298 30096 leader_election.cc:334] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
@ 0x7f32d5817daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 08:18:30.083897 27115 catalog_manager.cc:5484] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 1 to 2, leader changed from b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 2 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:30.086098 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d122f68d36024614a017e8cc3a1224c1" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:30.086269 30025 raft_consensus.cc:2454] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:30.086625 30093 leader_election.cc:302] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters: b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:30.087529 3170 raft_consensus.cc:2781] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:18:30.087603 3170 raft_consensus.cc:482] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.087663 3170 raft_consensus.cc:3037] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:30.090226 3170 raft_consensus.cc:504] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:30.090801 3170 leader_election.cc:288] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 election: Requested vote from peers 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:30.091459 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d122f68d36024614a017e8cc3a1224c1" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:30.091627 30025 raft_consensus.cc:3037] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Advancing to term 2
@ 0x7f32d5942480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 08:18:30.098086 3412 raft_consensus.cc:482] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.098245 3412 raft_consensus.cc:504] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:30.098912 3412 leader_election.cc:288] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:30.101646 30025 raft_consensus.cc:2454] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 2.
I20240502 08:18:30.101881 2943 raft_consensus.cc:482] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.102020 2943 raft_consensus.cc:504] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
@ 0x7f32d597aa8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 08:18:30.103456 30093 leader_election.cc:302] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
I20240502 08:18:30.103729 3170 raft_consensus.cc:2781] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader election won for term 2
W20240502 08:18:30.104111 30358 leader_election.cc:334] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.104434 3170 raft_consensus.cc:686] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [term 2 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:30.104719 3170 consensus_queue.cc:237] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:30.105588 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abc11a238a542e4982452b5ce341101" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
W20240502 08:18:30.107450 30096 leader_election.cc:334] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:30.107509 29961 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
@ 0x7f32d595d981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240502 08:18:30.107780 2943 leader_election.cc:288] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:30.108871 30358 leader_election.cc:302] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442
I20240502 08:18:30.109285 3412 raft_consensus.cc:2726] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:30.110601 29964 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.110757 29964 leader_election.cc:302] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:30.112114 3196 raft_consensus.cc:482] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.112284 3196 raft_consensus.cc:504] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:30.113025 3196 leader_election.cc:288] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
@ 0x7f32d595d70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 08:18:30.116158 27114 catalog_manager.cc:5484] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 1 to 2, leader changed from b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 2 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 08:18:30.116765 27326 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:30.117884 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.119104 3055 raft_consensus.cc:2726] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
@ 0x7f32c63b6b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f32c63ba19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f32c63ba019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 08:18:30.126801 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bea60d096f9408c9f4a857f8824d6a2" candidate_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:30.128916 27323 leader_election.cc:302] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83e171a5e16c4008923f1d43fc5ce7b1; no voters: 1f0bf4bc3c5146539146b825b048bff3, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:30.129278 3164 raft_consensus.cc:482] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.129340 3196 raft_consensus.cc:2726] T 0bea60d096f9408c9f4a857f8824d6a2 P 83e171a5e16c4008923f1d43fc5ce7b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:30.129443 3164 raft_consensus.cc:504] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:30.130208 3164 leader_election.cc:288] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
@ 0x7f32c639501e kudu::Thread::SuperviseThread() at ??:0
W20240502 08:18:30.133752 30096 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
@ 0x7f32c971a6db start_thread at ??:0
@ 0x7f32c255871f clone at ??:0
I20240502 08:18:30.176481 2975 raft_consensus.cc:482] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.176645 2975 raft_consensus.cc:504] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:30.177366 2975 leader_election.cc:288] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:30.178143 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abc11a238a542e4982452b5ce341101" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:30.178361 30418 raft_consensus.cc:2454] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:30.181327 27584 leader_election.cc:302] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:30.183838 2735 raft_consensus.cc:2781] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:18:30.183974 2735 raft_consensus.cc:482] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.184077 2735 raft_consensus.cc:3037] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:30.186739 2735 raft_consensus.cc:504] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:30.187307 2735 leader_election.cc:288] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: Requested vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:30.187453 2735 raft_consensus.cc:482] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.187558 2735 raft_consensus.cc:504] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:30.188154 2735 leader_election.cc:288] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:30.189985 27587 leader_election.cc:334] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.191053 3055 raft_consensus.cc:482] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.191112 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abc11a238a542e4982452b5ce341101" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:30.191273 30418 raft_consensus.cc:3037] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:30.191188 3055 raft_consensus.cc:504] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
W20240502 08:18:30.191342 27587 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:30.191480 27587 leader_election.cc:334] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.191761 3055 leader_election.cc:288] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:30.192611 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82841ff10f24cf7a0cf490be3defe69" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:30.193797 30418 raft_consensus.cc:2454] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 2.
W20240502 08:18:30.194226 29964 leader_election.cc:334] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.194557 29964 leader_election.cc:302] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442
W20240502 08:18:30.194952 27584 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.195077 27584 leader_election.cc:302] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:30.195320 2975 raft_consensus.cc:2726] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:30.195484 27584 leader_election.cc:302] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, bf8ac71b15f740a0a928911360773442; no voters: b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:30.195758 2975 raft_consensus.cc:2781] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:30.195873 2975 raft_consensus.cc:686] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [term 2 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:18:30.195915 3055 raft_consensus.cc:2726] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:30.196137 2975 consensus_queue.cc:237] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:30.200635 27114 catalog_manager.cc:5484] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 1 to 2, leader changed from b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 2 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:30.205127 3170 raft_consensus.cc:482] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.205238 3170 raft_consensus.cc:504] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:30.205870 3170 leader_election.cc:288] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
W20240502 08:18:30.207067 30096 leader_election.cc:334] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.207224 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5345b931504cfdbb272189b719efa4" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:30.207417 30418 raft_consensus.cc:2454] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:30.207780 30093 leader_election.cc:302] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters: b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:30.207995 3170 raft_consensus.cc:2781] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:18:30.208065 3170 raft_consensus.cc:482] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.208128 3170 raft_consensus.cc:3037] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:30.209887 3170 raft_consensus.cc:504] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:30.210425 3170 leader_election.cc:288] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:30.211045 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5345b931504cfdbb272189b719efa4" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
W20240502 08:18:30.211539 30096 leader_election.cc:334] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.211252 30418 raft_consensus.cc:3037] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:30.214213 30418 raft_consensus.cc:2454] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 2.
I20240502 08:18:30.215404 30093 leader_election.cc:302] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters: b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:30.215593 3170 raft_consensus.cc:2781] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:30.215703 3170 raft_consensus.cc:686] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [term 2 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:30.215919 3170 consensus_queue.cc:237] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:30.222352 27114 catalog_manager.cc:5484] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 1 to 2, leader changed from b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 2 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:30.234014 3434 raft_consensus.cc:482] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.234164 3434 raft_consensus.cc:504] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:30.234800 3434 leader_election.cc:288] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:30.238104 27194 leader_election.cc:334] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:30.240217 27191 leader_election.cc:334] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.240352 27191 leader_election.cc:302] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:30.240741 3434 raft_consensus.cc:2726] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:30.250653 2975 raft_consensus.cc:482] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.250833 2975 raft_consensus.cc:504] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:30.251421 2975 leader_election.cc:288] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:30.251983 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefcc470084f4ec9be6ba45d3ba22ce5" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:30.252175 30159 raft_consensus.cc:2454] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 2.
I20240502 08:18:30.252908 27587 leader_election.cc:302] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:30.253417 3433 raft_consensus.cc:2781] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:18:30.253491 3433 raft_consensus.cc:482] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.253582 3433 raft_consensus.cc:3037] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:30.256899 3433 raft_consensus.cc:504] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:30.257956 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefcc470084f4ec9be6ba45d3ba22ce5" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:30.258158 30159 raft_consensus.cc:3037] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:30.258208 3492 raft_consensus.cc:482] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.258378 3492 raft_consensus.cc:504] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:30.259167 3492 leader_election.cc:288] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:30.260685 30159 raft_consensus.cc:2454] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 3.
I20240502 08:18:30.261477 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08245cda17c44a77bcde2f864d2045ec" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
W20240502 08:18:30.264864 27587 leader_election.cc:334] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.265462 27587 leader_election.cc:302] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:30.265753 3492 raft_consensus.cc:2781] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:18:30.265918 3492 raft_consensus.cc:686] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [term 3 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:18:30.266219 3492 consensus_queue.cc:237] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
W20240502 08:18:30.267390 27584 leader_election.cc:334] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.267532 27584 leader_election.cc:302] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:30.267810 2975 raft_consensus.cc:2726] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:30.268338 3433 leader_election.cc:288] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 3 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:30.277815 27114 catalog_manager.cc:5484] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 2 to 3, leader changed from b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 3 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 08:18:30.287227 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.295636 3433 raft_consensus.cc:482] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.295819 3433 raft_consensus.cc:504] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:30.296577 3433 leader_election.cc:288] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:30.297049 27257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4b822c4aca4f7f92b73cc9279ddec5" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
W20240502 08:18:30.297216 27587 leader_election.cc:334] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.297257 27257 raft_consensus.cc:2454] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:30.297633 27586 leader_election.cc:302] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:30.297875 2954 raft_consensus.cc:482] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.298035 2954 raft_consensus.cc:504] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:30.298661 2954 leader_election.cc:288] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:30.298803 27587 leader_election.cc:334] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.299520 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cb28bc9e264d4882d482499b950349" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:30.299778 3433 raft_consensus.cc:2781] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:18:30.299847 3433 raft_consensus.cc:482] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.299926 3433 raft_consensus.cc:3037] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:30.300462 30096 leader_election.cc:334] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.301963 3433 raft_consensus.cc:504] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:30.302523 3433 leader_election.cc:288] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:30.302824 30096 leader_election.cc:302] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: b01e7c8a302d41078fa39cfef665ff59, bf8ac71b15f740a0a928911360773442
I20240502 08:18:30.303511 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4b822c4aca4f7f92b73cc9279ddec5" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:30.303719 27255 raft_consensus.cc:3037] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:30.307022 27255 raft_consensus.cc:2454] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 2.
I20240502 08:18:30.307389 2954 raft_consensus.cc:2726] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:30.308056 27586 leader_election.cc:302] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:30.308360 3433 raft_consensus.cc:2781] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:30.308544 3433 raft_consensus.cc:686] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [term 2 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
W20240502 08:18:30.308560 27587 leader_election.cc:334] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.308868 3433 consensus_queue.cc:237] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:30.314159 27114 catalog_manager.cc:5484] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 1 to 2, leader changed from b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 2 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:30.346020 3434 raft_consensus.cc:482] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.346195 3434 raft_consensus.cc:504] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:30.346804 3434 leader_election.cc:288] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:30.347457 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757f65cd77bf49d882c2545673282e03" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
W20240502 08:18:30.348189 27191 leader_election.cc:334] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.348304 27191 leader_election.cc:302] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:30.348675 3434 raft_consensus.cc:2726] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:30.350736 3055 raft_consensus.cc:482] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.350883 3055 raft_consensus.cc:504] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:30.351512 3055 leader_election.cc:288] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:30.353163 29964 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:30.354267 29961 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.354385 29961 leader_election.cc:302] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:30.354775 3055 raft_consensus.cc:2726] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:30.376013 2735 raft_consensus.cc:482] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.376174 2735 raft_consensus.cc:504] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:30.376781 2735 leader_election.cc:288] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:30.377475 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82841ff10f24cf7a0cf490be3defe69" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:30.377660 30025 raft_consensus.cc:2454] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:30.378131 27584 leader_election.cc:302] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, bf8ac71b15f740a0a928911360773442; no voters:
W20240502 08:18:30.378264 27587 leader_election.cc:334] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.378494 2735 raft_consensus.cc:2781] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:18:30.378563 2735 raft_consensus.cc:482] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.378618 2735 raft_consensus.cc:3037] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:30.381105 2735 raft_consensus.cc:504] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:30.382143 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82841ff10f24cf7a0cf490be3defe69" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:30.382321 30025 raft_consensus.cc:3037] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:30.382920 27587 leader_election.cc:334] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.384147 30025 raft_consensus.cc:2454] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 2.
I20240502 08:18:30.384495 27584 leader_election.cc:302] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, bf8ac71b15f740a0a928911360773442; no voters: b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:30.384727 3492 raft_consensus.cc:2781] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:30.384832 3492 raft_consensus.cc:686] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [term 2 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:18:30.385052 3492 consensus_queue.cc:237] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:30.386711 3055 raft_consensus.cc:482] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.386857 3055 raft_consensus.cc:504] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:30.387351 3055 leader_election.cc:288] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:30.388792 29964 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.388976 3433 raft_consensus.cc:482] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.389101 3433 raft_consensus.cc:504] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:30.389555 3433 leader_election.cc:288] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:30.390297 3433 raft_consensus.cc:482] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.390252 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ac899c820a43e4ab57454e224edc83" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:30.390405 3433 raft_consensus.cc:504] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
W20240502 08:18:30.390781 27584 leader_election.cc:334] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.390877 27584 leader_election.cc:302] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:30.390906 3433 leader_election.cc:288] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:30.391655 3433 raft_consensus.cc:2726] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:30.391837 2735 leader_election.cc:288] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: Requested vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
W20240502 08:18:30.392520 27584 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:30.393980 29963 connection.cc:535] client connection to 127.20.207.2:46291 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:30.395100 30095 connection.cc:535] client connection to 127.20.207.2:46291 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:18:30.395401 27114 catalog_manager.cc:5484] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 1 to 2, leader changed from b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 2 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:30.396296 3312 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:18:30.394536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.207.2:46291 (local address 127.20.207.5:40211)
0502 08:18:30.394678 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0502 08:18:30.394697 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0502 08:18:30.394805 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:18:30.396165 (+ 1360us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":48}
W20240502 08:18:30.396442 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:30.396477 27193 connection.cc:535] client connection to 127.20.207.2:46291 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:30.396535 29963 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:18:30.396603 29963 leader_election.cc:302] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
W20240502 08:18:30.396682 27586 connection.cc:535] client connection to 127.20.207.2:46291 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:30.396875 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:30.397316 30357 connection.cc:535] client connection to 127.20.207.2:46291 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:30.397593 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:30.397687 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:30.397761 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:30.397826 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.398245 2943 raft_consensus.cc:2726] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:30.398238 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:30.398346 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:30.398435 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:30.398499 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:30.399075 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:30.399168 27586 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:30.399235 27586 leader_election.cc:302] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
W20240502 08:18:30.399478 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:30.399731 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:30.399863 30095 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:30.399955 30095 leader_election.cc:302] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:30.400157 2954 raft_consensus.cc:2726] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:30.400241 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:30.400328 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:30.400378 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.400568 3492 raft_consensus.cc:2726] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:30.400701 3492 raft_consensus.cc:482] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.400787 3492 raft_consensus.cc:504] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:30.401211 3492 leader_election.cc:288] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:30.401870 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cb28bc9e264d4882d482499b950349" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:30.402091 30159 raft_consensus.cc:2454] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:30.402407 27587 leader_election.cc:302] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:30.402623 3492 raft_consensus.cc:2781] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:18:30.402704 3492 raft_consensus.cc:482] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:30.402765 3492 raft_consensus.cc:3037] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:30.402817 27587 leader_election.cc:334] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.404724 3492 raft_consensus.cc:504] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:30.405941 30159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cb28bc9e264d4882d482499b950349" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:30.406121 30159 raft_consensus.cc:3037] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:30.407718 27587 leader_election.cc:334] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.408612 30159 raft_consensus.cc:2454] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 2.
I20240502 08:18:30.408766 3492 leader_election.cc:288] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:30.409634 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.409695 27587 leader_election.cc:302] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6, bf8ac71b15f740a0a928911360773442; no voters: b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:30.409922 3492 raft_consensus.cc:2781] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:30.410044 3492 raft_consensus.cc:686] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [term 2 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:18:30.410301 3492 consensus_queue.cc:237] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:30.413938 3434 raft_consensus.cc:482] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.414121 3434 raft_consensus.cc:504] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:30.414755 3434 leader_election.cc:288] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:30.418181 27191 leader_election.cc:334] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:30.418285 27193 leader_election.cc:334] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:30.418347 27191 leader_election.cc:302] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:30.418689 3434 raft_consensus.cc:2726] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:30.417852 27114 catalog_manager.cc:5484] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 1 to 2, leader changed from b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3) to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 2 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:30.429009 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.448544 3055 raft_consensus.cc:482] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.448660 3055 raft_consensus.cc:504] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:30.449184 3055 leader_election.cc:288] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:30.450264 29961 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:30.450381 29963 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:30.450481 29963 leader_election.cc:302] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:30.450667 3055 raft_consensus.cc:2726] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:30.453716 2954 raft_consensus.cc:482] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.453830 2954 raft_consensus.cc:504] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:30.454337 2954 leader_election.cc:288] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:30.454728 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757f65cd77bf49d882c2545673282e03" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:30.454921 27255 raft_consensus.cc:2454] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:30.455210 30095 leader_election.cc:302] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 84cda20738d2452da166e31fc6da57a6; no voters:
W20240502 08:18:30.455327 30093 leader_election.cc:334] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.455400 2954 raft_consensus.cc:2781] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:18:30.455467 2954 raft_consensus.cc:482] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.455525 2954 raft_consensus.cc:3037] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:30.457619 2954 raft_consensus.cc:504] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:30.458156 2954 leader_election.cc:288] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:30.458595 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757f65cd77bf49d882c2545673282e03" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:30.458761 27255 raft_consensus.cc:3037] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:30.459276 30093 leader_election.cc:334] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.460906 27255 raft_consensus.cc:2454] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 2.
I20240502 08:18:30.461212 30095 leader_election.cc:302] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 84cda20738d2452da166e31fc6da57a6; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:30.461573 2954 raft_consensus.cc:2781] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:30.461704 2954 raft_consensus.cc:686] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [term 2 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:30.461933 2954 consensus_queue.cc:237] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
W20240502 08:18:30.464188 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:30.465479 27114 catalog_manager.cc:5484] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 1 to 2, leader changed from 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 2 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:30.470822 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.475402 3412 raft_consensus.cc:482] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.475519 3412 raft_consensus.cc:504] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:30.476042 3412 leader_election.cc:288] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:30.476464 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2776f857a5f47e9b8022639dc54757d" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
W20240502 08:18:30.477309 30355 leader_election.cc:334] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.477412 30355 leader_election.cc:302] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, bf8ac71b15f740a0a928911360773442
I20240502 08:18:30.477614 3412 raft_consensus.cc:2726] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:18:30.480036 3492 raft_consensus.cc:482] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.480203 3492 raft_consensus.cc:504] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:30.480710 3492 leader_election.cc:288] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:30.481906 27586 leader_election.cc:334] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:30.482040 27584 leader_election.cc:334] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.482113 27586 leader_election.cc:302] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:30.482350 3492 raft_consensus.cc:2726] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:30.491067 3055 raft_consensus.cc:482] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.491189 3055 raft_consensus.cc:504] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:30.491663 3055 leader_election.cc:288] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:30.492661 29961 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:30.492789 29964 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:30.492879 29964 leader_election.cc:302] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:30.493042 3055 raft_consensus.cc:2726] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:30.520378 3492 raft_consensus.cc:482] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.520506 3492 raft_consensus.cc:504] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:30.520958 3492 leader_election.cc:288] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:30.522414 27587 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:30.522609 27584 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.522768 27584 leader_election.cc:302] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:30.522992 3492 raft_consensus.cc:2726] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:30.545154 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.547626 3492 raft_consensus.cc:482] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.547776 3492 raft_consensus.cc:504] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:30.548465 3492 leader_election.cc:288] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:30.549122 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2776f857a5f47e9b8022639dc54757d" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:30.549314 30418 raft_consensus.cc:2454] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 2.
W20240502 08:18:30.549784 27584 leader_election.cc:334] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.550030 27584 leader_election.cc:302] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:30.550582 3492 raft_consensus.cc:2781] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:18:30.550686 3492 raft_consensus.cc:482] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.550781 3492 raft_consensus.cc:3037] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:30.552758 3492 raft_consensus.cc:504] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:30.553229 3492 leader_election.cc:288] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 3 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:30.553769 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2776f857a5f47e9b8022639dc54757d" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:30.553939 30418 raft_consensus.cc:3037] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Advancing to term 3
W20240502 08:18:30.554311 27584 leader_election.cc:334] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.555697 30418 raft_consensus.cc:2454] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 3.
I20240502 08:18:30.556030 27584 leader_election.cc:302] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:30.556213 3492 raft_consensus.cc:2781] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:18:30.556330 3492 raft_consensus.cc:686] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [term 3 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:18:30.556566 3492 consensus_queue.cc:237] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } }
I20240502 08:18:30.559948 27114 catalog_manager.cc:5484] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 2 to 3, leader changed from 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 3 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:30.570822 2954 raft_consensus.cc:482] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.571007 2954 raft_consensus.cc:504] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:30.571650 2954 leader_election.cc:288] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:30.573050 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ac899c820a43e4ab57454e224edc83" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:30.573253 27652 raft_consensus.cc:2454] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
W20240502 08:18:30.573561 30093 leader_election.cc:334] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.573709 30096 leader_election.cc:302] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6, bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4
I20240502 08:18:30.574019 2954 raft_consensus.cc:2781] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:18:30.574105 2954 raft_consensus.cc:482] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.574180 2954 raft_consensus.cc:3037] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:30.576244 2954 raft_consensus.cc:504] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:30.576759 2954 leader_election.cc:288] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 election: Requested vote from peers bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:30.577173 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ac899c820a43e4ab57454e224edc83" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:30.577356 27652 raft_consensus.cc:3037] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:30.579949 27652 raft_consensus.cc:2454] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 2.
I20240502 08:18:30.580256 30096 leader_election.cc:302] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:30.580495 2954 raft_consensus.cc:2781] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader election won for term 2
W20240502 08:18:30.580858 30093 leader_election.cc:334] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.580883 2954 raft_consensus.cc:686] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [term 2 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:30.581130 2954 consensus_queue.cc:237] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:30.583096 3266 raft_consensus.cc:482] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.583267 3266 raft_consensus.cc:504] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:30.583818 3266 leader_election.cc:288] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:30.585892 30357 leader_election.cc:334] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:30.586333 30355 leader_election.cc:334] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.586421 30355 leader_election.cc:302] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:30.586624 3435 raft_consensus.cc:2726] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:30.589232 27114 catalog_manager.cc:5484] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 1 to 2, leader changed from 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 2 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:30.597455 30418 raft_consensus.cc:1261] T eb5345b931504cfdbb272189b719efa4 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:30.598026 3170 consensus_queue.cc:1035] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:30.601845 27652 raft_consensus.cc:1261] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:18:30.602659 3055 consensus_queue.cc:1035] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:18:30.603471 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.603749 30025 raft_consensus.cc:1261] T d122f68d36024614a017e8cc3a1224c1 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:18:30.604311 3434 raft_consensus.cc:482] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.604311 3170 consensus_queue.cc:1035] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:30.604470 3434 raft_consensus.cc:504] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:30.605176 3434 leader_election.cc:288] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:30.605953 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08245cda17c44a77bcde2f864d2045ec" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:30.606161 27650 raft_consensus.cc:2454] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 1.
I20240502 08:18:30.606535 27194 leader_election.cc:302] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:30.606951 3434 raft_consensus.cc:2781] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:18:30.607036 3434 raft_consensus.cc:482] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.607110 3434 raft_consensus.cc:3037] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:30.608661 27191 leader_election.cc:334] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.609670 3434 raft_consensus.cc:504] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:30.610702 27650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08245cda17c44a77bcde2f864d2045ec" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442"
I20240502 08:18:30.610869 27650 raft_consensus.cc:3037] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:30.612897 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.613186 27650 raft_consensus.cc:2454] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 in term 2.
I20240502 08:18:30.613576 27194 leader_election.cc:302] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:30.613809 3238 raft_consensus.cc:2781] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader election won for term 2
W20240502 08:18:30.619170 27191 leader_election.cc:334] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.619724 3434 leader_election.cc:288] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 election: Requested vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
W20240502 08:18:30.633585 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.654264 3238 raft_consensus.cc:686] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 LEADER]: Becoming Leader. State: Replica: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, State: Running, Role: LEADER
I20240502 08:18:30.654606 3238 consensus_queue.cc:237] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:30.658210 3170 raft_consensus.cc:482] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:30.658335 3170 raft_consensus.cc:504] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:30.658972 3170 leader_election.cc:288] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:18:30.660120 27115 catalog_manager.cc:5484] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 reported cstate change: term changed from 1 to 2, leader changed from 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7) to 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1). New cstate: current_term: 2 leader_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:30.661921 30095 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:30.662639 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:30.662717 30093 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:30.662803 30093 leader_election.cc:302] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:30.663143 3170 raft_consensus.cc:2726] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:18:30.679191 27103 catalog_manager.cc:4573] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6cf739b314d84db08bc4cb32b0e5b0f4: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:30.697696 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.720603 30159 raft_consensus.cc:1261] T fefcc470084f4ec9be6ba45d3ba22ce5 P 84cda20738d2452da166e31fc6da57a6 [term 3 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240502 08:18:30.721094 3492 consensus_queue.cc:1035] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 08:18:30.751585 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.769158 30418 raft_consensus.cc:1261] T 2abc11a238a542e4982452b5ce341101 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:18:30.769605 3433 consensus_queue.cc:1035] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:18:30.776007 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:30.836570 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.867442 30025 raft_consensus.cc:1261] T b82841ff10f24cf7a0cf490be3defe69 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:18:30.867862 2735 consensus_queue.cc:1035] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 08:18:30.871704 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.882572 27255 raft_consensus.cc:1261] T 757f65cd77bf49d882c2545673282e03 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:18:30.883091 3164 consensus_queue.cc:1035] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:18:30.890491 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:30.905851 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:30.906044 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.934581 27255 raft_consensus.cc:1261] T de4b822c4aca4f7f92b73cc9279ddec5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:18:30.935154 2975 consensus_queue.cc:1035] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:30.939008 30418 raft_consensus.cc:1261] T c2776f857a5f47e9b8022639dc54757d P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 3 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 08:18:30.940346 3433 consensus_queue.cc:1035] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 08:18:30.947039 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.998606 30156 raft_consensus.cc:1261] T 56cb28bc9e264d4882d482499b950349 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:18:30.999083 2975 consensus_queue.cc:1035] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:31.035343 27652 raft_consensus.cc:1261] T 65ac899c820a43e4ab57454e224edc83 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:18:31.035826 2954 consensus_queue.cc:1035] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:18:31.039812 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:31.039911 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:31.051000 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:31.149223 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:31.151134 27652 raft_consensus.cc:1261] T 08245cda17c44a77bcde2f864d2045ec P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Refusing update from remote peer 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:18:31.151690 3238 consensus_queue.cc:1035] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:18:31.157714 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:31.172988 3238 raft_consensus.cc:482] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.173170 3238 raft_consensus.cc:504] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:31.173864 3238 leader_election.cc:288] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
I20240502 08:18:31.176005 3238 raft_consensus.cc:482] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.176121 3238 raft_consensus.cc:504] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
W20240502 08:18:31.176515 27193 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.176771 3238 leader_election.cc:288] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
W20240502 08:18:31.178030 27193 leader_election.cc:334] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.178282 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8fe72edcef41a28fdeea86d501f93b" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
W20240502 08:18:31.178751 27194 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:31.178857 27194 leader_election.cc:302] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:31.178948 27191 leader_election.cc:302] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:31.179080 3238 raft_consensus.cc:2726] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:31.179221 3238 raft_consensus.cc:2726] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:31.183537 3055 raft_consensus.cc:482] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.183669 3055 raft_consensus.cc:504] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:31.184024 3164 raft_consensus.cc:482] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.184275 3055 leader_election.cc:288] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:31.184176 3164 raft_consensus.cc:504] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:31.184721 3164 leader_election.cc:288] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:31.184998 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a26e01d42b429694641f0ceca11265" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
I20240502 08:18:31.185137 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf739b314d84db08bc4cb32b0e5b0f4" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
W20240502 08:18:31.186350 29963 leader_election.cc:334] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.186452 29963 leader_election.cc:302] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442
I20240502 08:18:31.186658 3055 raft_consensus.cc:2726] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:31.187290 30095 leader_election.cc:334] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.187793 30093 leader_election.cc:302] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 1f0bf4bc3c5146539146b825b048bff3, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:31.188088 3164 raft_consensus.cc:2726] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:31.190032 3055 raft_consensus.cc:482] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.190143 3055 raft_consensus.cc:504] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:31.190650 3055 leader_election.cc:288] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:31.191757 29963 leader_election.cc:334] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:31.191946 29964 leader_election.cc:334] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:31.192021 29964 leader_election.cc:302] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:31.192195 3055 raft_consensus.cc:2726] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:31.193799 3433 raft_consensus.cc:482] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.193945 3433 raft_consensus.cc:504] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:31.194507 3433 leader_election.cc:288] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:31.196578 27586 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:31.196710 27587 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:31.196790 27587 leader_election.cc:302] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:31.196946 3433 raft_consensus.cc:2726] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:31.221107 3266 raft_consensus.cc:482] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.221274 3266 raft_consensus.cc:504] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:31.221987 3266 leader_election.cc:288] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:31.222761 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8fe72edcef41a28fdeea86d501f93b" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:31.222990 27255 raft_consensus.cc:2454] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 1.
I20240502 08:18:31.223383 30357 leader_election.cc:302] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters:
I20240502 08:18:31.223620 3266 raft_consensus.cc:2781] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:18:31.223690 3266 raft_consensus.cc:482] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.223758 3266 raft_consensus.cc:3037] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:31.224081 30357 leader_election.cc:334] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.225642 3266 raft_consensus.cc:504] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:31.226116 3266 leader_election.cc:288] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505)
I20240502 08:18:31.226706 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8fe72edcef41a28fdeea86d501f93b" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:31.226850 27255 raft_consensus.cc:3037] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:31.227224 30357 leader_election.cc:334] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.227367 3266 raft_consensus.cc:482] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.227480 3266 raft_consensus.cc:504] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:31.228101 3266 leader_election.cc:288] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:31.228587 30156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a0a58eb52f41bf818e5102996d09e0" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:31.229142 27255 raft_consensus.cc:2454] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 2.
W20240502 08:18:31.229280 30357 leader_election.cc:334] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.229382 30357 leader_election.cc:302] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, 84cda20738d2452da166e31fc6da57a6
I20240502 08:18:31.229619 3266 raft_consensus.cc:2726] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:31.229802 30357 leader_election.cc:302] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:31.230043 3266 raft_consensus.cc:2781] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:31.230167 3266 raft_consensus.cc:686] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:18:31.230391 3266 consensus_queue.cc:237] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } }
I20240502 08:18:31.233953 27115 catalog_manager.cc:5484] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 1 to 2, leader changed from 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 2 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:31.245564 3164 raft_consensus.cc:482] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.245705 3164 raft_consensus.cc:504] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:31.246259 3164 leader_election.cc:288] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:31.246686 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a0a58eb52f41bf818e5102996d09e0" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
I20240502 08:18:31.246882 30418 raft_consensus.cc:2454] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 1.
I20240502 08:18:31.247191 30093 leader_election.cc:302] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters:
W20240502 08:18:31.247390 30095 leader_election.cc:334] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.247639 3164 raft_consensus.cc:2781] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:18:31.247695 3164 raft_consensus.cc:482] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.247762 3164 raft_consensus.cc:3037] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:31.249367 3164 raft_consensus.cc:504] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:31.249806 3164 leader_election.cc:288] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619)
I20240502 08:18:31.250247 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a0a58eb52f41bf818e5102996d09e0" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad"
I20240502 08:18:31.250413 30418 raft_consensus.cc:3037] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:31.250932 30095 leader_election.cc:334] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.252197 30418 raft_consensus.cc:2454] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cda20738d2452da166e31fc6da57a6 in term 2.
I20240502 08:18:31.252487 30093 leader_election.cc:302] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 84cda20738d2452da166e31fc6da57a6; no voters: 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:31.252727 3164 raft_consensus.cc:2781] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:31.252911 3164 raft_consensus.cc:686] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [term 2 LEADER]: Becoming Leader. State: Replica: 84cda20738d2452da166e31fc6da57a6, State: Running, Role: LEADER
I20240502 08:18:31.253147 3164 consensus_queue.cc:237] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:31.256781 27115 catalog_manager.cc:5484] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 reported cstate change: term changed from 1 to 2, leader changed from 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) to 84cda20738d2452da166e31fc6da57a6 (127.20.207.6). New cstate: current_term: 2 leader_uuid: "84cda20738d2452da166e31fc6da57a6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:31.269280 3164 raft_consensus.cc:482] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.269392 3164 raft_consensus.cc:504] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:31.269816 3164 leader_election.cc:288] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), bf8ac71b15f740a0a928911360773442 (127.20.207.4:33309)
I20240502 08:18:31.270228 27652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "459e9f5bf6d24ba9b36686d49ff12e2b" candidate_uuid: "84cda20738d2452da166e31fc6da57a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bf8ac71b15f740a0a928911360773442" is_pre_election: true
W20240502 08:18:31.270948 30095 leader_election.cc:334] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.271031 30095 leader_election.cc:302] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, bf8ac71b15f740a0a928911360773442
I20240502 08:18:31.271234 3164 raft_consensus.cc:2726] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:31.273073 3266 raft_consensus.cc:482] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.273211 3266 raft_consensus.cc:504] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:31.273648 3266 leader_election.cc:288] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:31.274994 30358 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:31.275144 30357 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.275238 30357 leader_election.cc:302] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:31.275413 3266 raft_consensus.cc:2726] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:31.365640 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.438421 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.483309 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:31.490342 3492 raft_consensus.cc:482] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.490504 3492 raft_consensus.cc:504] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:31.491134 3492 leader_election.cc:288] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:31.491665 30025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a26e01d42b429694641f0ceca11265" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3" is_pre_election: true
I20240502 08:18:31.491854 30025 raft_consensus.cc:2454] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:31.492230 27584 leader_election.cc:302] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, bf8ac71b15f740a0a928911360773442; no voters:
W20240502 08:18:31.492368 27586 leader_election.cc:334] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.492429 3492 raft_consensus.cc:2781] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:18:31.492482 3492 raft_consensus.cc:482] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.492565 3492 raft_consensus.cc:3037] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:31.494375 3492 raft_consensus.cc:504] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:31.494868 3492 leader_election.cc:288] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5:37815)
I20240502 08:18:31.495366 30028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a26e01d42b429694641f0ceca11265" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1f0bf4bc3c5146539146b825b048bff3"
I20240502 08:18:31.495527 30028 raft_consensus.cc:3037] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:31.497673 30028 raft_consensus.cc:2454] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 2.
I20240502 08:18:31.497989 27584 leader_election.cc:302] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:31.498234 3492 raft_consensus.cc:2781] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:31.498337 3492 raft_consensus.cc:686] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [term 2 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:18:31.498580 3492 consensus_queue.cc:237] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:31.498952 3434 raft_consensus.cc:482] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
W20240502 08:18:31.499050 27586 leader_election.cc:334] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.499096 3434 raft_consensus.cc:504] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:31.499723 3434 leader_election.cc:288] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8:41619), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:31.500541 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9b191ce9234c028b990efd74a59e2e" candidate_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" is_pre_election: true
W20240502 08:18:31.502130 27193 leader_election.cc:334] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.502221 27193 leader_election.cc:302] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:31.502415 3434 raft_consensus.cc:2726] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:18:31.502700 27115 catalog_manager.cc:5484] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 1 to 2, leader changed from 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 2 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:31.525596 3433 raft_consensus.cc:482] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.525753 3433 raft_consensus.cc:504] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:31.526414 3433 leader_election.cc:288] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:31.527809 27586 leader_election.cc:334] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:31.529767 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.530563 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:31.531266 27584 leader_election.cc:334] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:31.531378 27584 leader_election.cc:302] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:31.531579 3433 raft_consensus.cc:2726] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:31.551218 3433 raft_consensus.cc:482] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.551330 3433 raft_consensus.cc:504] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:31.551820 3433 leader_election.cc:288] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:31.552166 30156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "459e9f5bf6d24ba9b36686d49ff12e2b" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:31.552357 30156 raft_consensus.cc:2454] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 1.
I20240502 08:18:31.552690 27587 leader_election.cc:302] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6, bf8ac71b15f740a0a928911360773442; no voters:
I20240502 08:18:31.552932 3433 raft_consensus.cc:2781] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:18:31.553011 3433 raft_consensus.cc:482] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.553087 3433 raft_consensus.cc:3037] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:31.554100 27586 leader_election.cc:334] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.555037 3433 raft_consensus.cc:504] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:31.555440 3433 leader_election.cc:288] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: Requested vote from peers 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:31.555809 30156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "459e9f5bf6d24ba9b36686d49ff12e2b" candidate_uuid: "bf8ac71b15f740a0a928911360773442" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:31.555965 30156 raft_consensus.cc:3037] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:31.556512 27586 leader_election.cc:334] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.557840 30156 raft_consensus.cc:2454] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf8ac71b15f740a0a928911360773442 in term 2.
I20240502 08:18:31.558149 27587 leader_election.cc:302] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6, bf8ac71b15f740a0a928911360773442; no voters: 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:31.558351 3433 raft_consensus.cc:2781] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:31.558487 3433 raft_consensus.cc:686] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [term 2 LEADER]: Becoming Leader. State: Replica: bf8ac71b15f740a0a928911360773442, State: Running, Role: LEADER
I20240502 08:18:31.558682 3433 consensus_queue.cc:237] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:31.563362 27115 catalog_manager.cc:5484] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 reported cstate change: term changed from 1 to 2, leader changed from 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) to bf8ac71b15f740a0a928911360773442 (127.20.207.4). New cstate: current_term: 2 leader_uuid: "bf8ac71b15f740a0a928911360773442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } health_report { overall_health: HEALTHY } } }
W20240502 08:18:31.565217 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.567108 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.567258 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:31.571585 2281 raft_consensus.cc:482] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.571704 2281 raft_consensus.cc:504] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:31.572242 2281 leader_election.cc:288] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:31.572613 30156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf739b314d84db08bc4cb32b0e5b0f4" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6" is_pre_election: true
I20240502 08:18:31.572819 30156 raft_consensus.cc:2454] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 1.
I20240502 08:18:31.573202 29964 leader_election.cc:302] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters:
W20240502 08:18:31.573621 29963 leader_election.cc:334] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.573977 2281 raft_consensus.cc:2781] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:18:31.574105 2281 raft_consensus.cc:482] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.574185 2281 raft_consensus.cc:3037] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:31.575959 2281 raft_consensus.cc:504] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:31.576373 2281 leader_election.cc:288] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 election: Requested vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 84cda20738d2452da166e31fc6da57a6 (127.20.207.6:38899)
I20240502 08:18:31.576752 30156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf739b314d84db08bc4cb32b0e5b0f4" candidate_uuid: "1f0bf4bc3c5146539146b825b048bff3" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "84cda20738d2452da166e31fc6da57a6"
I20240502 08:18:31.576893 30156 raft_consensus.cc:3037] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:31.577395 29963 leader_election.cc:334] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.578619 30156 raft_consensus.cc:2454] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0bf4bc3c5146539146b825b048bff3 in term 2.
I20240502 08:18:31.578923 29964 leader_election.cc:302] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3, 84cda20738d2452da166e31fc6da57a6; no voters: 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:31.579190 2281 raft_consensus.cc:2781] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:31.579303 2281 raft_consensus.cc:686] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 LEADER]: Becoming Leader. State: Replica: 1f0bf4bc3c5146539146b825b048bff3, State: Running, Role: LEADER
I20240502 08:18:31.579545 2281 consensus_queue.cc:237] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:31.583781 27115 catalog_manager.cc:5484] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 reported cstate change: term changed from 1 to 2, leader changed from 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) to 1f0bf4bc3c5146539146b825b048bff3 (127.20.207.5). New cstate: current_term: 2 leader_uuid: "1f0bf4bc3c5146539146b825b048bff3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:31.584568 3266 raft_consensus.cc:482] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.584730 3266 raft_consensus.cc:504] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:31.585887 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9b191ce9234c028b990efd74a59e2e" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" is_pre_election: true
I20240502 08:18:31.586145 27255 raft_consensus.cc:2454] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 2.
I20240502 08:18:31.586587 30357 leader_election.cc:302] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters:
I20240502 08:18:31.586861 3435 raft_consensus.cc:2781] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:18:31.586944 3435 raft_consensus.cc:482] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Starting leader election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:31.587007 3435 raft_consensus.cc:3037] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:18:31.586624 3266 leader_election.cc:288] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:31.588191 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.588289 30357 leader_election.cc:334] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.589179 3435 raft_consensus.cc:504] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:31.589584 3435 leader_election.cc:288] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 3 election: Requested vote from peers 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 (127.20.207.1:38505), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
I20240502 08:18:31.589900 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9b191ce9234c028b990efd74a59e2e" candidate_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60"
I20240502 08:18:31.590096 27255 raft_consensus.cc:3037] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Advancing to term 3
W20240502 08:18:31.590755 30357 leader_election.cc:334] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:31.592028 27255 raft_consensus.cc:2454] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a1e33dcdbf42f58ea72ce1c0afb0ad in term 3.
I20240502 08:18:31.592346 30357 leader_election.cc:302] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60, 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:31.592561 3435 raft_consensus.cc:2781] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:18:31.592679 3435 raft_consensus.cc:686] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 3 LEADER]: Becoming Leader. State: Replica: 31a1e33dcdbf42f58ea72ce1c0afb0ad, State: Running, Role: LEADER
I20240502 08:18:31.592921 3435 consensus_queue.cc:237] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
W20240502 08:18:31.593425 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:31.596506 27115 catalog_manager.cc:5484] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad reported cstate change: term changed from 2 to 3, leader changed from 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2) to 31a1e33dcdbf42f58ea72ce1c0afb0ad (127.20.207.8). New cstate: current_term: 3 leader_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:31.603811 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:31.623687 3238 raft_consensus.cc:482] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:31.623824 3238 raft_consensus.cc:504] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:31.624348 3238 leader_election.cc:288] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:31.625962 27193 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:31.626164 27194 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:31.626257 27194 leader_election.cc:302] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:31.626457 3238 raft_consensus.cc:2726] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:18:31.640570 27255 raft_consensus.cc:1261] T bc8fe72edcef41a28fdeea86d501f93b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:18:31.641357 3435 consensus_queue.cc:1035] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:31.641917 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.653051 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:31.653548 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.664104 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.674827 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:31.694950 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.695235 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.704021 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.704792 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:31.709086 3433 raft_consensus.cc:482] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:31.709239 3433 raft_consensus.cc:504] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:31.709854 3433 leader_election.cc:288] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:31.711319 27587 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:31.711464 27584 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:31.711547 27584 leader_election.cc:302] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:31.711830 3433 raft_consensus.cc:2726] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:31.729562 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.738948 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.743201 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:31.748407 27103 catalog_manager.cc:4573] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6cf739b314d84db08bc4cb32b0e5b0f4: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:31.771044 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.779557 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:31.860244 30418 raft_consensus.cc:1261] T 04a0a58eb52f41bf818e5102996d09e0 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 2 FOLLOWER]: Refusing update from remote peer 84cda20738d2452da166e31fc6da57a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:31.860749 3170 consensus_queue.cc:1035] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 08:18:31.872673 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.875952 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.887665 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:31.888199 3433 raft_consensus.cc:482] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:31.888321 3433 raft_consensus.cc:504] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:31.888857 3433 leader_election.cc:288] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:31.890059 27587 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:31.890434 27584 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:31.890574 27584 leader_election.cc:302] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:31.890856 3433 raft_consensus.cc:2726] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:31.897107 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.905441 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:31.905534 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.910100 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.933969 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.970705 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:31.972874 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:31.978346 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:32.006347 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:32.007896 3660 raft_consensus.cc:482] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:32.008028 3660 raft_consensus.cc:504] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:32.008519 3660 leader_election.cc:288] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:32.009727 30095 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:32.009910 30096 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:32.010083 30096 leader_election.cc:302] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:32.010285 3660 raft_consensus.cc:2726] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:32.020097 30025 raft_consensus.cc:1261] T b6a26e01d42b429694641f0ceca11265 P 1f0bf4bc3c5146539146b825b048bff3 [term 2 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:18:32.020541 3433 consensus_queue.cc:1035] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:18:32.022030 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:32.026111 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:32.030172 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:32.040656 27255 raft_consensus.cc:1261] T 2e9b191ce9234c028b990efd74a59e2e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 3 FOLLOWER]: Refusing update from remote peer 31a1e33dcdbf42f58ea72ce1c0afb0ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240502 08:18:32.041426 3266 consensus_queue.cc:1035] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 08:18:32.045718 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:32.048269 30156 raft_consensus.cc:1261] T 459e9f5bf6d24ba9b36686d49ff12e2b P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Refusing update from remote peer bf8ac71b15f740a0a928911360773442: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:18:32.048760 3433 consensus_queue.cc:1035] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:32.064469 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:32.093745 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:32.096063 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:32.101058 3433 raft_consensus.cc:482] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:32.101233 3433 raft_consensus.cc:504] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
W20240502 08:18:32.101495 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:32.101869 3433 leader_election.cc:288] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:32.103058 27584 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:32.103263 27586 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:32.103374 27586 leader_election.cc:302] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:32.103698 3684 raft_consensus.cc:2726] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:32.112016 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:32.128728 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:32.132622 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:32.146327 3055 raft_consensus.cc:482] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:32.146494 3055 raft_consensus.cc:504] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:32.147112 3055 leader_election.cc:288] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:32.149134 29964 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:32.149310 29963 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:32.149425 29963 leader_election.cc:302] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:32.149621 2281 raft_consensus.cc:2726] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:32.177870 30156 raft_consensus.cc:1261] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Refusing update from remote peer 1f0bf4bc3c5146539146b825b048bff3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:32.178337 2281 consensus_queue.cc:1035] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 08:18:32.181911 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:32.211683 3711 raft_consensus.cc:482] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:32.211848 3711 raft_consensus.cc:504] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:32.212944 3711 leader_election.cc:288] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:32.213794 29961 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:32.214124 29964 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:32.214212 29964 leader_election.cc:302] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:32.215167 3055 raft_consensus.cc:2726] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:32.263926 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:32.283299 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:32.287307 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:32.289019 3711 raft_consensus.cc:482] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:32.289149 3711 raft_consensus.cc:504] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:32.289743 3711 leader_election.cc:288] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:32.291020 29961 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:32.291154 29964 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:32.291246 29964 leader_election.cc:302] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:32.291455 3711 raft_consensus.cc:2726] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:32.323457 3687 raft_consensus.cc:482] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:32.323593 3687 raft_consensus.cc:504] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:32.324142 3687 leader_election.cc:288] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:32.325392 27191 leader_election.cc:334] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:32.325531 27194 leader_election.cc:334] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:32.325657 27194 leader_election.cc:302] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:32.325908 3687 raft_consensus.cc:2726] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:32.344494 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:32.350658 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:32.365346 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:32.403360 3711 raft_consensus.cc:482] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:32.403523 3711 raft_consensus.cc:504] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:32.404132 3711 leader_election.cc:288] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:32.405879 29963 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:32.406280 29961 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:32.406414 29961 leader_election.cc:302] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:32.406669 3711 raft_consensus.cc:2726] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:32.435418 3660 raft_consensus.cc:482] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:32.435593 3660 raft_consensus.cc:504] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:32.436770 3660 leader_election.cc:288] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:32.437644 30093 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:32.437790 30095 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:32.437935 30095 leader_election.cc:302] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:32.438179 3660 raft_consensus.cc:2726] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:18:32.496726 3412 raft_consensus.cc:482] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:32.496876 3412 raft_consensus.cc:504] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:32.497331 3412 leader_election.cc:288] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:32.498530 30355 leader_election.cc:334] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:32.498642 30357 leader_election.cc:334] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:32.498715 30355 leader_election.cc:302] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:32.498929 3412 raft_consensus.cc:2726] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:32.524643 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:32.543530 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:32.554265 3433 raft_consensus.cc:482] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:32.554454 3433 raft_consensus.cc:504] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:32.555086 3433 leader_election.cc:288] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:32.556324 27584 leader_election.cc:334] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:32.556515 27586 leader_election.cc:334] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:32.556667 27586 leader_election.cc:302] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:32.557070 3433 raft_consensus.cc:2726] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:32.601454 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:32.604496 3055 raft_consensus.cc:482] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:32.604661 3055 raft_consensus.cc:504] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:32.605122 3055 leader_election.cc:288] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:32.606320 29961 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:32.606508 29964 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:32.606634 29964 leader_election.cc:302] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:32.606837 3055 raft_consensus.cc:2726] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:32.611410 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:32.632460 3687 raft_consensus.cc:482] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:32.632613 3687 raft_consensus.cc:504] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:32.633175 3687 leader_election.cc:288] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:32.634562 27191 leader_election.cc:334] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:32.634728 27193 leader_election.cc:334] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:32.634845 27193 leader_election.cc:302] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:32.635154 3687 raft_consensus.cc:2726] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:32.644169 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:32.646826 3055 raft_consensus.cc:482] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:32.647058 3055 raft_consensus.cc:504] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:32.648294 3055 leader_election.cc:288] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:32.648918 29963 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:32.649936 29964 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:32.650144 29964 leader_election.cc:302] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:32.650512 3055 raft_consensus.cc:2726] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:32.656519 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:32.719476 3433 raft_consensus.cc:482] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:32.719678 3433 raft_consensus.cc:504] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:32.720351 3433 leader_election.cc:288] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:32.721763 27584 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:32.722034 27587 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:32.722164 27587 leader_election.cc:302] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:32.722400 3433 raft_consensus.cc:2726] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:32.727828 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:32.744340 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:32.754351 3687 raft_consensus.cc:482] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:32.754555 3687 raft_consensus.cc:504] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:32.755252 3687 leader_election.cc:288] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:32.756095 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:32.757025 27193 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:32.758502 27194 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:32.759084 27194 leader_election.cc:302] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:32.759533 3687 raft_consensus.cc:2726] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:32.763537 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:32.772603 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:32.808636 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:32.842900 3055 raft_consensus.cc:482] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:32.843107 3055 raft_consensus.cc:504] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:32.843845 3055 leader_election.cc:288] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:32.845165 29964 leader_election.cc:334] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:32.845395 29963 leader_election.cc:334] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:32.845497 29963 leader_election.cc:302] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:32.845778 3055 raft_consensus.cc:2726] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:32.860914 3433 raft_consensus.cc:482] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:32.861142 3433 raft_consensus.cc:504] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:32.861716 3433 leader_election.cc:288] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:32.862921 27586 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:32.863134 27587 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:32.863231 27587 leader_election.cc:302] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:32.863462 3433 raft_consensus.cc:2726] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:32.896370 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:32.931075 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:32.938722 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:32.945250 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:32.974429 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:33.037403 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:33.049198 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:33.083370 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:33.148347 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:33.182830 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:33.220072 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:33.240243 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:33.242937 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:33.250137 3412 raft_consensus.cc:482] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:33.250255 3412 raft_consensus.cc:504] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:33.250803 3412 leader_election.cc:288] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:33.251803 30357 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:33.252067 30358 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:33.252141 30358 leader_election.cc:302] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:33.252306 3412 raft_consensus.cc:2726] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:33.272854 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:33.293794 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:33.299144 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:33.300513 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:33.323503 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:33.326285 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:33.349843 3774 raft_consensus.cc:482] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:33.350050 3774 raft_consensus.cc:504] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:33.350760 3774 leader_election.cc:288] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:33.352195 27587 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:33.352494 27584 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:33.352625 27584 leader_election.cc:302] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:33.352941 3774 raft_consensus.cc:2726] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:33.360512 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:33.369818 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:33.412777 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:33.435094 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:33.467203 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:33.475185 3774 raft_consensus.cc:482] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:33.475379 3774 raft_consensus.cc:504] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:33.476085 3774 leader_election.cc:288] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:33.477264 27586 leader_election.cc:334] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:33.477399 27584 leader_election.cc:334] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:33.477523 27584 leader_election.cc:302] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:33.477782 3774 raft_consensus.cc:2726] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:33.614427 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:33.650218 3433 raft_consensus.cc:482] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:33.650390 3433 raft_consensus.cc:504] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:33.651072 3433 leader_election.cc:288] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:33.652242 27586 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:33.653316 27584 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:33.653445 27584 leader_election.cc:302] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:33.653666 3433 raft_consensus.cc:2726] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:33.655323 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:33.689113 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:33.844259 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:33.848235 27103 catalog_manager.cc:4573] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6cf739b314d84db08bc4cb32b0e5b0f4: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:33.938041 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:18:33.946305 3687 raft_consensus.cc:482] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:33.946456 3687 raft_consensus.cc:504] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:33.947073 3687 leader_election.cc:288] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:33.948173 27194 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:33.948323 27193 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:33.948503 27193 leader_election.cc:302] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:33.948730 3687 raft_consensus.cc:2726] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:18:33.956101 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:33.974707 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:33.978037 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:33.982939 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:33.998929 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.018734 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:34.061127 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:34.107992 3711 raft_consensus.cc:482] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:34.108193 3711 raft_consensus.cc:504] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:34.108736 3711 leader_election.cc:288] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:34.110193 29964 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:34.110636 29961 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:34.110761 29961 leader_election.cc:302] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:34.111020 3711 raft_consensus.cc:2726] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:34.115538 3711 raft_consensus.cc:482] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:34.115690 3711 raft_consensus.cc:504] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:34.116192 3711 leader_election.cc:288] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:34.117393 29964 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:34.117688 29963 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:34.117803 29963 leader_election.cc:302] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:34.118062 3711 raft_consensus.cc:2726] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:34.134222 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.147799 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.165774 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.167892 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:18:34.173744 3880 raft_consensus.cc:482] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:34.173902 3880 raft_consensus.cc:504] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:34.174517 3880 leader_election.cc:288] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:34.175562 27587 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:34.175711 27584 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:34.175820 27584 leader_election.cc:302] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:34.176003 3880 raft_consensus.cc:2726] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:34.183220 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.186887 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.199889 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.225641 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.233053 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.243382 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.280207 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.280454 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:34.287207 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.304334 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.313331 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.320147 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:34.321043 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.326444 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.326752 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.333607 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.355684 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:34.363411 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.405524 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:34.422389 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.435544 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:34.465060 29963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 162 similar messages]
W20240502 08:18:34.465497 27191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 192 similar messages]
I20240502 08:18:34.469709 3711 raft_consensus.cc:482] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:34.469866 3711 raft_consensus.cc:504] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:34.470373 3711 leader_election.cc:288] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:34.471472 29964 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:34.471622 29961 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:34.471733 29961 leader_election.cc:302] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:34.471936 3711 raft_consensus.cc:2726] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:34.475153 30093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 225 similar messages]
W20240502 08:18:34.476459 30355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240502 08:18:34.477205 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.489694 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:34.505620 27587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 192 similar messages]
I20240502 08:18:34.506395 3687 raft_consensus.cc:482] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:34.506511 3687 raft_consensus.cc:504] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:34.507064 3687 leader_election.cc:288] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:34.508237 27194 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:34.508509 27193 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:34.508608 27193 leader_election.cc:302] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:34.508836 3687 raft_consensus.cc:2726] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:34.510766 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.523149 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.526240 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:18:34.536756 3687 raft_consensus.cc:482] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:34.536916 3687 raft_consensus.cc:504] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:34.537429 3687 leader_election.cc:288] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:34.538578 27191 leader_election.cc:334] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:34.538753 27193 leader_election.cc:334] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:34.538839 27193 leader_election.cc:302] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:34.539063 3687 raft_consensus.cc:2726] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:34.622139 3880 raft_consensus.cc:482] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
W20240502 08:18:34.622246 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:18:34.622323 3880 raft_consensus.cc:504] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:34.622948 3880 leader_election.cc:288] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:34.623957 27584 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:34.624080 27587 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:34.624213 27587 leader_election.cc:302] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:34.624409 3880 raft_consensus.cc:2726] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:34.650658 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.662240 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:18:34.673763 3687 raft_consensus.cc:482] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:34.673923 3687 raft_consensus.cc:504] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
W20240502 08:18:34.675987 27194 leader_election.cc:334] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:34.674592 3687 leader_election.cc:288] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:34.676872 27191 leader_election.cc:334] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:34.676987 27191 leader_election.cc:302] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:34.677206 3687 raft_consensus.cc:2726] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:34.696331 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.698706 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.699368 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.701653 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:34.755712 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.760157 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:18:34.769959 3660 raft_consensus.cc:482] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:34.770113 3660 raft_consensus.cc:504] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:34.770669 3660 leader_election.cc:288] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:34.771762 30095 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:34.772167 30096 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:34.772238 30096 leader_election.cc:302] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:34.772433 3660 raft_consensus.cc:2726] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:34.794426 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:18:34.798655 3433 raft_consensus.cc:482] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:34.798844 3433 raft_consensus.cc:504] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:34.799414 3433 leader_election.cc:288] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:34.800565 27584 leader_election.cc:334] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:34.800720 27586 leader_election.cc:334] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:34.800838 27586 leader_election.cc:302] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:34.801112 3433 raft_consensus.cc:2726] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:34.805580 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.811599 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.839835 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:34.857601 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.918748 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:34.940356 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:34.984326 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:34.988209 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:35.001983 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:35.033320 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:35.127359 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:35.134217 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:35.181114 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:35.226445 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:18:35.304867 3433 raft_consensus.cc:482] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:35.305040 3433 raft_consensus.cc:504] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:35.305699 3433 leader_election.cc:288] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:35.307215 27584 leader_election.cc:334] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:35.307410 27586 leader_election.cc:334] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:35.307536 27586 leader_election.cc:302] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:35.307788 3433 raft_consensus.cc:2726] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:35.353708 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:35.357708 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:35.371392 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:35.372437 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:18:35.380501 3055 raft_consensus.cc:482] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:35.380658 3055 raft_consensus.cc:504] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:35.381212 3055 leader_election.cc:288] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:35.382465 29963 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:35.387264 29964 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:35.387387 29964 leader_election.cc:302] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:35.387681 3055 raft_consensus.cc:2726] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:35.408022 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:35.430835 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:18:35.431416 3435 raft_consensus.cc:482] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:35.431574 3435 raft_consensus.cc:504] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:35.432195 3435 leader_election.cc:288] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:35.433630 30357 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:35.434176 30358 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:35.434306 30358 leader_election.cc:302] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:35.434551 3435 raft_consensus.cc:2726] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:35.471941 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:35.483122 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:35.503674 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:35.559931 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:35.591225 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:35.592239 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:18:35.617957 3660 raft_consensus.cc:482] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:35.618137 3660 raft_consensus.cc:504] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:35.618783 3660 leader_election.cc:288] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:35.619899 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:35.620306 30093 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:35.620476 30095 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:35.620573 30095 leader_election.cc:302] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:35.620810 3660 raft_consensus.cc:2726] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:18:35.634666 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:18:35.676549 3433 raft_consensus.cc:482] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:35.676776 3433 raft_consensus.cc:504] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:35.677398 3433 leader_election.cc:288] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:35.679055 27586 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:35.679191 27584 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:35.679312 27584 leader_election.cc:302] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:35.679576 3433 raft_consensus.cc:2726] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:35.726089 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:18:35.726913 4010 raft_consensus.cc:482] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:35.727075 4010 raft_consensus.cc:504] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:35.728484 4010 leader_election.cc:288] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:35.728893 27586 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:35.729674 27587 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:35.729820 27587 leader_election.cc:302] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:35.730067 4010 raft_consensus.cc:2726] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:35.742288 3055 raft_consensus.cc:482] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:35.742437 3055 raft_consensus.cc:504] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:35.742971 3055 leader_election.cc:288] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:35.744463 29961 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:35.745082 29963 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:35.745165 29963 leader_election.cc:302] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
W20240502 08:18:35.745170 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:18:35.745354 3055 raft_consensus.cc:2726] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:35.747536 4011 raft_consensus.cc:482] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:35.747653 4011 raft_consensus.cc:504] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:35.748159 4011 leader_election.cc:288] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:35.749815 30355 leader_election.cc:334] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:35.749989 30357 leader_election.cc:334] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:35.750097 30357 leader_election.cc:302] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:35.750531 4011 raft_consensus.cc:2726] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:35.763548 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:18:35.771937 3055 raft_consensus.cc:482] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:35.772089 3055 raft_consensus.cc:504] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:35.772693 3055 leader_election.cc:288] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:35.773962 29964 leader_election.cc:334] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:35.774102 29963 leader_election.cc:334] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:35.774209 29963 leader_election.cc:302] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:35.774421 3055 raft_consensus.cc:2726] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:35.781212 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:35.784796 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:35.791668 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:35.792500 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:35.814677 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:35.863457 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:18:35.885973 3055 raft_consensus.cc:482] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:35.886150 3055 raft_consensus.cc:504] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:35.886695 3055 leader_election.cc:288] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:35.887708 29964 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:35.887941 29963 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:35.888094 29963 leader_election.cc:302] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:35.888355 3055 raft_consensus.cc:2726] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:35.924687 3055 raft_consensus.cc:482] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:35.924878 3055 raft_consensus.cc:504] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:35.925511 3055 leader_election.cc:288] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:35.926966 29964 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:35.927094 29961 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:35.927213 29961 leader_election.cc:302] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:35.927438 3055 raft_consensus.cc:2726] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:35.958334 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:35.966691 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:35.989501 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:36.159202 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:36.188221 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:36.211053 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:36.211205 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.211674 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:36.238991 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.287153 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:36.373957 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.508117 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.527356 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.528245 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:18:36.550642 3711 raft_consensus.cc:482] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:36.550828 3711 raft_consensus.cc:504] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:36.551473 3711 leader_election.cc:288] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:36.552553 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.553040 29961 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:36.558463 29964 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:36.558574 29964 leader_election.cc:302] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:36.558791 3711 raft_consensus.cc:2726] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:36.570065 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.588451 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.599045 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.649533 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:36.666643 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:36.673807 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.691045 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.711583 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:36.722254 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.726194 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.734864 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.737946 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.738783 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:36.746171 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.748836 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.755838 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:36.759709 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.764210 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:18:36.778123 4012 raft_consensus.cc:482] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:36.778270 4012 raft_consensus.cc:504] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:36.780087 4012 leader_election.cc:288] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:36.780223 27191 leader_election.cc:334] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:36.780858 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.781409 27193 leader_election.cc:334] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:36.781495 27193 leader_election.cc:302] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:36.781756 4012 raft_consensus.cc:2726] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:36.785918 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.790343 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:18:36.800740 4010 raft_consensus.cc:482] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:36.800868 4010 raft_consensus.cc:504] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:36.801370 4010 leader_election.cc:288] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:36.802558 27584 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:36.802780 27587 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:36.802867 27587 leader_election.cc:302] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:36.803064 4010 raft_consensus.cc:2726] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:36.827457 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:36.843214 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.845160 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.863871 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:18:36.876132 3660 raft_consensus.cc:482] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:36.876294 3660 raft_consensus.cc:504] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:36.876963 3660 leader_election.cc:288] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:36.879349 30096 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:36.879676 30095 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:36.879781 30095 leader_election.cc:302] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:36.880167 3660 raft_consensus.cc:2726] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:36.882637 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:36.919194 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:36.928743 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.050868 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.053207 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.062625 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.095403 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.103803 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:37.149622 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.160300 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.171873 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.193217 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.240994 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:18:37.265141 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:37.265451 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:37.306757 4010 raft_consensus.cc:482] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:37.306913 4010 raft_consensus.cc:504] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:37.307483 4010 leader_election.cc:288] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:37.308643 27587 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:37.308764 27584 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:37.308866 27584 leader_election.cc:302] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:37.309047 4010 raft_consensus.cc:2726] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:37.315704 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.330216 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.332062 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.344354 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.349102 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:18:37.419807 4010 raft_consensus.cc:482] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:37.419993 4010 raft_consensus.cc:504] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:37.420522 4010 leader_election.cc:288] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:37.421573 27586 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:37.421675 27587 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:37.421794 27587 leader_election.cc:302] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:37.422019 4010 raft_consensus.cc:2726] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:37.455117 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.469152 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.471206 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.493587 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.511502 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.580598 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.601490 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:37.604941 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.606266 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.652853 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.750041 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.769088 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:37.773805 3711 raft_consensus.cc:482] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:37.773919 3711 raft_consensus.cc:504] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:37.774585 3711 leader_election.cc:288] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:37.775924 29964 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:37.776093 29961 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:37.776173 29961 leader_election.cc:302] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:37.776371 3711 raft_consensus.cc:2726] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:37.803189 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:37.825687 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.837405 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.855455 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:18:37.857103 4138 raft_consensus.cc:482] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:37.857240 4138 raft_consensus.cc:504] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:37.857826 4138 leader_election.cc:288] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:37.860507 30357 leader_election.cc:334] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:37.863305 30355 leader_election.cc:334] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:37.863446 30355 leader_election.cc:302] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:37.863818 4138 raft_consensus.cc:2726] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:37.869982 4012 raft_consensus.cc:482] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:37.870147 4012 raft_consensus.cc:504] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:37.870700 4012 leader_election.cc:288] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:37.872550 27193 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:37.873003 27194 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:37.873137 27194 leader_election.cc:302] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:37.873347 4012 raft_consensus.cc:2726] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:37.912882 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.932245 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:37.966670 27103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240502 08:18:37.967705 27103 catalog_manager.cc:4573] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6cf739b314d84db08bc4cb32b0e5b0f4: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:38.000770 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:38.011848 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:38.027312 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:38.129782 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:18:38.173957 4138 raft_consensus.cc:482] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:38.174157 4138 raft_consensus.cc:504] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:38.174652 4138 leader_election.cc:288] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:38.175683 30357 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:38.175845 30358 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:38.175951 30358 leader_election.cc:302] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:38.176154 4138 raft_consensus.cc:2726] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:38.191354 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:38.201714 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:38.218164 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:38.230140 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:18:38.243750 3055 raft_consensus.cc:482] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:38.243956 3055 raft_consensus.cc:504] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:38.244536 3055 leader_election.cc:288] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:38.246228 29961 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:38.246423 29963 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:38.246570 29963 leader_election.cc:302] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:38.247053 3711 raft_consensus.cc:2726] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:38.311285 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:38.314033 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:38.314416 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:38.320592 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:38.349284 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:38.374622 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:38.433606 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:38.452697 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:18:38.455214 4010 raft_consensus.cc:482] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:38.455351 4010 raft_consensus.cc:504] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:38.456000 4010 leader_election.cc:288] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:38.457096 27584 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:38.457895 27587 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:38.457976 27587 leader_election.cc:302] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:38.458189 4010 raft_consensus.cc:2726] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:38.459669 4012 raft_consensus.cc:482] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:38.459823 4012 raft_consensus.cc:504] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:38.460310 4012 leader_election.cc:288] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:38.461434 27194 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:38.461625 27193 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:38.461732 27193 leader_election.cc:302] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:38.461927 4012 raft_consensus.cc:2726] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:18:38.471680 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:18:38.486254 4012 raft_consensus.cc:482] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:38.486411 4012 raft_consensus.cc:504] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:38.487063 4012 leader_election.cc:288] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:38.488286 27194 leader_election.cc:334] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:38.488744 27191 leader_election.cc:334] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:38.488829 27191 leader_election.cc:302] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:38.489050 4012 raft_consensus.cc:2726] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:38.501900 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:38.807783 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:38.812137 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:38.815162 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:38.830317 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:38.862983 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:18:38.871017 3055 raft_consensus.cc:482] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:38.871183 3055 raft_consensus.cc:504] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:38.872304 3055 leader_election.cc:288] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:38.873399 29961 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:38.873589 29964 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:38.873683 29964 leader_election.cc:302] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:38.873842 3055 raft_consensus.cc:2726] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:38.877511 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:38.912487 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:38.919835 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:38.943746 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:38.962344 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.000123 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.014848 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.065268 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.074693 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.099380 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.113057 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.148394 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.179402 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.187422 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.207422 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.219732 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.232720 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:39.236511 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.236610 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.238181 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.240178 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.254491 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.279829 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:39.287772 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.288949 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:39.310827 4139 raft_consensus.cc:482] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:39.310992 4139 raft_consensus.cc:504] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:39.311491 4139 leader_election.cc:288] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:39.312703 27586 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:39.312858 27584 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:39.312973 27584 leader_election.cc:302] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:39.313172 4139 raft_consensus.cc:2726] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:39.342633 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:39.345399 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.361933 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:39.368916 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.377331 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.412011 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.429431 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:39.444437 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.446784 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:39.466430 29964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240502 08:18:39.467159 27191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 177 similar messages]
W20240502 08:18:39.467535 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.477568 30095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 218 similar messages]
W20240502 08:18:39.488751 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:18:39.505751 4315 raft_consensus.cc:482] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:39.505908 4315 raft_consensus.cc:504] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:39.506722 4315 leader_election.cc:288] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:39.507766 30095 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:39.507884 30093 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:39.507992 30093 leader_election.cc:302] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:39.508193 4315 raft_consensus.cc:2726] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:18:39.512373 3055 raft_consensus.cc:482] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:39.512493 3055 raft_consensus.cc:504] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:39.513041 3055 leader_election.cc:288] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:39.514371 29964 leader_election.cc:334] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:39.514518 29963 leader_election.cc:334] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:39.514636 29963 leader_election.cc:302] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:39.514853 3055 raft_consensus.cc:2726] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:39.527551 30357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240502 08:18:39.528254 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:39.538592 27587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 180 similar messages]
I20240502 08:18:39.576560 4010 raft_consensus.cc:482] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:39.576730 4010 raft_consensus.cc:504] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:39.577243 4010 leader_election.cc:288] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:39.577765 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.578275 27586 leader_election.cc:334] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:39.578447 27584 leader_election.cc:334] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:39.578531 27584 leader_election.cc:302] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:39.578688 4010 raft_consensus.cc:2726] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:39.584147 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.640890 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.672976 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.695940 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:39.703420 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.713588 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.741290 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:18:39.746388 4010 raft_consensus.cc:482] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:39.746559 4010 raft_consensus.cc:504] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:39.747229 4010 leader_election.cc:288] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:39.748498 27586 leader_election.cc:334] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:39.748610 27584 leader_election.cc:334] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:39.748692 27584 leader_election.cc:302] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:39.748869 4010 raft_consensus.cc:2726] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:39.781100 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.822551 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:18:39.828106 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:39.843411 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.866851 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.926645 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:39.928193 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.941449 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.959427 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:39.989831 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.080114 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:18:40.093602 4294 raft_consensus.cc:482] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:40.093786 4294 raft_consensus.cc:504] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:40.094462 4294 leader_election.cc:288] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:40.094542 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.095345 29963 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:40.095795 29964 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:40.095903 29964 leader_election.cc:302] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:40.096115 4294 raft_consensus.cc:2726] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:40.140148 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.166728 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.236815 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.253031 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:40.266203 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.288774 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.325152 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:40.330729 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.375842 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.384958 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.407732 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.414327 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:40.480266 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.557335 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.612118 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.626462 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.649355 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.652382 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.665542 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.671120 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.678510 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:40.786216 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:40.812783 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:40.824295 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:40.872439 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:40.912158 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:18:40.961325 4273 raft_consensus.cc:482] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:40.961485 4273 raft_consensus.cc:504] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:40.962127 4273 leader_election.cc:288] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:40.963274 27194 leader_election.cc:334] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:40.963446 27191 leader_election.cc:334] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:40.963582 27191 leader_election.cc:302] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:40.963795 4273 raft_consensus.cc:2726] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:40.986519 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:41.005041 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:41.021620 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:41.030907 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:41.060274 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:41.127350 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:41.208355 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.288621 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:41.296728 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.317144 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:41.368343 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:41.419840 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.503453 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.534479 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:18:41.537894 4139 raft_consensus.cc:482] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
W20240502 08:18:41.537990 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:18:41.538074 4139 raft_consensus.cc:504] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:41.538692 4139 leader_election.cc:288] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:41.539783 27584 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:41.541632 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.541815 27587 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:41.541904 27587 leader_election.cc:302] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:41.542127 4139 raft_consensus.cc:2726] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:41.554138 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.562889 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.564417 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.572355 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.602619 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.612977 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.626717 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.648172 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.671619 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:41.686183 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.687613 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.711416 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:41.717041 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:41.723619 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.743155 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.755017 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.758740 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.764330 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:41.764662 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.772853 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.791882 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:41.806895 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.811066 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:18:41.844457 4399 raft_consensus.cc:482] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:41.844594 4399 raft_consensus.cc:504] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:41.845193 4399 leader_election.cc:288] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:41.847496 30355 leader_election.cc:334] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:41.848121 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.848217 30357 leader_election.cc:334] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:41.848340 30357 leader_election.cc:302] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:41.848521 4399 raft_consensus.cc:2726] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:41.852981 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.865398 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.904271 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:41.912308 4294 raft_consensus.cc:482] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:41.912482 4294 raft_consensus.cc:504] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:41.913177 4294 leader_election.cc:288] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:41.915585 29964 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:41.916033 29961 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:41.916167 29961 leader_election.cc:302] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:41.916628 4294 raft_consensus.cc:2726] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:41.924116 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:41.933645 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:41.942610 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:18:41.946113 4421 raft_consensus.cc:482] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:41.946270 4421 raft_consensus.cc:504] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:41.946851 4421 leader_election.cc:288] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:41.947930 30095 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:41.948076 30096 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:41.948205 30096 leader_election.cc:302] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:41.948453 4421 raft_consensus.cc:2726] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:41.969906 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.049685 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:18:42.063956 4273 raft_consensus.cc:482] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:42.064107 4273 raft_consensus.cc:504] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:42.064607 4273 leader_election.cc:288] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:42.066140 27194 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:42.066285 27193 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:42.066411 27193 leader_election.cc:302] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:42.066618 4273 raft_consensus.cc:2726] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:18:42.092466 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:42.096081 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.116923 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.160349 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.176577 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:18:42.180373 4294 raft_consensus.cc:482] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:42.180507 4294 raft_consensus.cc:504] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:42.181234 4294 leader_election.cc:288] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:42.182432 29964 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:42.182771 29963 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:42.182857 29963 leader_election.cc:302] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:42.183058 4571 raft_consensus.cc:2726] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:42.190320 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:42.271360 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.273540 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.280150 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.283460 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:18:42.355417 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.380759 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.397964 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:42.409008 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.431890 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.445245 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:42.450291 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.466109 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.479597 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:18:42.573405 4139 raft_consensus.cc:482] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:42.573576 4139 raft_consensus.cc:504] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:42.574156 4139 leader_election.cc:288] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:42.575277 27584 leader_election.cc:334] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:42.575459 27586 leader_election.cc:334] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:42.575562 27586 leader_election.cc:302] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:42.575789 4139 raft_consensus.cc:2726] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:42.593423 4421 raft_consensus.cc:482] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:42.593559 4421 raft_consensus.cc:504] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:42.594137 4421 leader_election.cc:288] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:42.595237 30095 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:42.595542 30093 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:42.595630 30093 leader_election.cc:302] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:42.595870 4421 raft_consensus.cc:2726] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:18:42.602371 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.614089 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:42.615446 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:18:42.655731 4582 raft_consensus.cc:482] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:42.655874 4582 raft_consensus.cc:504] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:42.656461 4582 leader_election.cc:288] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:42.657838 27584 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:42.657953 27587 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:42.658082 27587 leader_election.cc:302] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:42.658447 4582 raft_consensus.cc:2726] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:42.686476 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.753839 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:18:42.785624 4528 raft_consensus.cc:482] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:42.785769 4528 raft_consensus.cc:504] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:42.786331 4528 leader_election.cc:288] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:42.787617 27193 leader_election.cc:334] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:42.787757 27191 leader_election.cc:334] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:42.787868 27191 leader_election.cc:302] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:42.788194 4528 raft_consensus.cc:2726] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:42.837940 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.838140 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:18:42.884037 4617 raft_consensus.cc:482] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:42.884229 4617 raft_consensus.cc:504] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:42.884840 4617 leader_election.cc:288] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:42.886893 29961 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:42.887321 29963 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:42.887410 29963 leader_election.cc:302] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:42.887616 4617 raft_consensus.cc:2726] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:42.906304 4617 raft_consensus.cc:482] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:42.906469 4617 raft_consensus.cc:504] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:42.907068 4617 leader_election.cc:288] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:42.908028 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:42.908121 29961 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:42.908851 29964 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:42.908943 29964 leader_election.cc:302] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:42.909332 4617 raft_consensus.cc:2726] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:42.909652 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.933728 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.938233 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:42.941817 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:18:42.981812 4615 raft_consensus.cc:482] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:42.981971 4615 raft_consensus.cc:504] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:42.982627 4615 leader_election.cc:288] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:42.984308 30358 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:42.984434 30357 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:42.984535 30357 leader_election.cc:302] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:42.984738 4615 raft_consensus.cc:2726] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:42.987006 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:42.989733 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:43.009500 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:43.069650 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:43.086272 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:43.127878 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:43.135758 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:43.193277 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:43.229220 4528 raft_consensus.cc:482] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:43.229346 4528 raft_consensus.cc:504] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:43.229853 4528 leader_election.cc:288] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:43.231169 27194 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:43.231619 27193 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:43.231704 27193 leader_election.cc:302] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:43.231913 4528 raft_consensus.cc:2726] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:43.236534 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:43.310386 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:43.327435 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:43.328083 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:43.328568 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:43.353353 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:43.419296 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:43.419675 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:43.519769 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:43.553633 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:43.571259 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:43.594379 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:43.647995 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:43.791939 4582 raft_consensus.cc:482] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:43.792105 4582 raft_consensus.cc:504] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:43.792762 4582 leader_election.cc:288] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:43.794167 27586 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:43.795543 27587 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:43.795648 27587 leader_election.cc:302] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:43.795853 4582 raft_consensus.cc:2726] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:43.831189 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:43.854384 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:43.916752 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:43.919005 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:43.924854 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:43.957019 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:43.973762 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:43.992846 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:43.992985 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:43.999330 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.004998 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.024713 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.032405 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:44.070662 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.071208 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.077333 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.105123 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.107266 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.138284 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.139992 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:44.184871 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:44.206979 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:44.219534 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.236482 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.244963 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.253629 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.257090 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:44.316160 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.316555 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:44.327025 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.353448 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:44.370870 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.381938 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.404897 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.405753 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:18:44.414044 4750 raft_consensus.cc:482] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:44.414178 4750 raft_consensus.cc:504] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:44.414988 4750 leader_election.cc:288] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:44.416003 27587 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:44.416584 27584 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:44.416668 27584 leader_election.cc:302] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:44.417304 4750 raft_consensus.cc:2726] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:44.428030 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.434039 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.460079 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.470889 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.481838 30095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 214 similar messages]
W20240502 08:18:44.490620 27193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 181 similar messages]
W20240502 08:18:44.532866 29964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 08:18:44.533610 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:44.553184 30358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240502 08:18:44.555712 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:18:44.560150 27587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 176 similar messages]
W20240502 08:18:44.577261 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.586225 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.623836 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:44.643747 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.651821 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.666514 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.673681 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.736295 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.752926 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.758586 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.795037 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.805691 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:18:44.817183 4617 raft_consensus.cc:482] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:44.817303 4617 raft_consensus.cc:504] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:44.817741 4617 leader_election.cc:288] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:44.818899 29963 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:44.819108 29964 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:44.819211 29964 leader_election.cc:302] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:44.819397 4617 raft_consensus.cc:2726] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:44.827407 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:44.863379 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.876139 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.920989 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.937970 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.965301 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:44.976300 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:45.035490 4617 raft_consensus.cc:482] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:45.035646 4617 raft_consensus.cc:504] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:45.036120 4617 leader_election.cc:288] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:45.037228 29964 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:45.037420 29961 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:45.037515 29961 leader_election.cc:302] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:45.037739 4617 raft_consensus.cc:2726] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:45.060600 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:45.081420 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:45.102519 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:18:45.130532 4661 raft_consensus.cc:482] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:45.130693 4661 raft_consensus.cc:504] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:45.131332 4661 leader_election.cc:288] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:45.134127 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:45.134789 30093 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:45.145975 30095 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:45.146132 30095 leader_election.cc:302] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:45.146744 4661 raft_consensus.cc:2726] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:18:45.200186 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:45.222101 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:45.253019 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:45.311805 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:45.331454 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:45.340324 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:45.416843 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:45.417028 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:45.432914 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:45.445737 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:45.472942 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:18:45.503858 4750 raft_consensus.cc:482] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:45.503993 4750 raft_consensus.cc:504] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:45.504520 4750 leader_election.cc:288] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:45.506063 27586 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:45.506219 27587 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:45.506368 27587 leader_election.cc:302] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:45.506593 4750 raft_consensus.cc:2726] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:45.543874 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:45.578692 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:45.596508 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:18:45.606844 4617 raft_consensus.cc:482] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:45.607029 4617 raft_consensus.cc:504] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:45.607542 4617 leader_election.cc:288] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:45.608672 29964 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:45.608788 29963 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:45.608906 29963 leader_election.cc:302] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:45.609114 4617 raft_consensus.cc:2726] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:45.611980 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:45.641158 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:18:45.641499 4617 raft_consensus.cc:482] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:45.641646 4617 raft_consensus.cc:504] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:45.642289 4617 leader_election.cc:288] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:45.643827 29964 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:45.643998 29961 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:45.644100 29961 leader_election.cc:302] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:45.644472 4617 raft_consensus.cc:2726] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:45.680357 4750 raft_consensus.cc:482] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:45.680555 4750 raft_consensus.cc:504] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:45.681221 4750 leader_election.cc:288] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:45.682658 27586 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:45.682895 27584 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:45.683058 27584 leader_election.cc:302] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:45.683296 4750 raft_consensus.cc:2726] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:45.691380 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:45.702066 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:45.750069 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:45.786749 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:45.800750 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:18:45.813728 4617 raft_consensus.cc:482] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:45.813906 4617 raft_consensus.cc:504] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:45.814576 4617 leader_election.cc:288] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:45.815990 29963 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:45.816129 29961 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:45.816226 29961 leader_election.cc:302] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:45.816479 4821 raft_consensus.cc:2726] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:45.921519 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:18:45.969632 4750 raft_consensus.cc:482] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:45.969820 4750 raft_consensus.cc:504] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:45.970635 4750 leader_election.cc:288] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:45.971917 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:45.971951 27584 leader_election.cc:334] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:45.972131 27586 leader_election.cc:334] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:45.972272 27586 leader_election.cc:302] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:45.972487 4750 raft_consensus.cc:2726] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:45.990217 4788 raft_consensus.cc:482] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:45.990358 4788 raft_consensus.cc:504] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:45.990912 4788 leader_election.cc:288] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:45.992419 27194 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:45.992826 27193 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:45.992924 27193 leader_election.cc:302] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:45.993130 4788 raft_consensus.cc:2726] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:18:45.993541 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:46.028805 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:46.042260 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:46.062958 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:46.095836 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:46.172852 27103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:46.173882 27103 catalog_manager.cc:4573] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6cf739b314d84db08bc4cb32b0e5b0f4: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:46.273545 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:46.310209 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.312750 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.391927 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.392520 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.394795 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.421830 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:18:46.422731 4924 raft_consensus.cc:482] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:46.422873 4924 raft_consensus.cc:504] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:46.423518 4924 leader_election.cc:288] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:46.425241 27586 leader_election.cc:334] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:46.425350 27584 leader_election.cc:334] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:46.425424 27584 leader_election.cc:302] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:46.427260 4924 raft_consensus.cc:2726] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:46.435997 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.444763 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:46.451144 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.455874 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:46.521659 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.565469 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:46.598413 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:46.606539 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.612159 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.627045 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.642949 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.649164 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:18:46.655189 4617 raft_consensus.cc:482] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:46.655319 4617 raft_consensus.cc:504] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:46.655853 4617 leader_election.cc:288] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:46.657291 29963 leader_election.cc:334] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:46.671051 29964 leader_election.cc:334] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:46.671154 29964 leader_election.cc:302] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:46.671331 4617 raft_consensus.cc:2726] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:46.675190 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:46.719694 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.726763 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:46.727655 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.730592 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:46.766759 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.780977 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.800663 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.802979 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:46.821877 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.825662 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.843223 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.861868 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:46.867687 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.911588 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.914780 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:18:46.915967 4266 raft_consensus.cc:482] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:46.916090 4266 raft_consensus.cc:504] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:46.916724 4266 leader_election.cc:288] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:46.918740 30358 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:46.927012 30357 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:46.927120 30357 leader_election.cc:302] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:46.927376 4266 raft_consensus.cc:2726] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:46.954679 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.956964 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:46.958923 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.962389 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.977550 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:46.988281 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.071228 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.086093 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.093976 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.103940 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.122740 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.166608 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.174136 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.175877 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.224678 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.243445 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.276490 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.279239 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:18:47.328706 4990 raft_consensus.cc:482] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:47.328845 4990 raft_consensus.cc:504] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:47.329339 4990 leader_election.cc:288] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:47.330427 27587 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:47.330637 27584 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:47.330713 27584 leader_election.cc:302] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:47.330896 4990 raft_consensus.cc:2726] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:47.349267 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:47.351445 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:47.367468 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.368924 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:18:47.380718 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.412700 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.458448 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.458712 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:47.473416 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:47.487545 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.553792 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.578174 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.646452 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.647562 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.777021 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:47.790122 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.844197 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.853216 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:47.935283 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:18:47.942461 4617 raft_consensus.cc:482] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:47.942589 4617 raft_consensus.cc:504] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:47.943156 4617 leader_election.cc:288] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:47.944144 29961 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:47.944252 29964 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:47.944363 29964 leader_election.cc:302] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:47.944573 4617 raft_consensus.cc:2726] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:47.948976 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:48.043102 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:48.080832 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:48.082335 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:48.100462 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:48.107374 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:48.123876 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:48.141237 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:48.157035 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:48.183292 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:18:48.227113 4990 raft_consensus.cc:482] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:48.227306 4990 raft_consensus.cc:504] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:48.227998 4990 leader_election.cc:288] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:48.229177 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:48.230160 27587 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:48.230563 27584 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:48.230661 27584 leader_election.cc:302] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:48.230847 4990 raft_consensus.cc:2726] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:48.339376 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:48.352506 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:48.361218 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:48.380787 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:18:48.390834 5034 raft_consensus.cc:482] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:48.390995 5034 raft_consensus.cc:504] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:48.391618 5034 leader_election.cc:288] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:48.393471 27191 leader_election.cc:334] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:48.393990 27193 leader_election.cc:334] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:48.394117 27193 leader_election.cc:302] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:48.394515 5034 raft_consensus.cc:2726] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:48.408440 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:48.471956 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:48.503038 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:48.648994 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:48.658849 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:48.701926 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:48.767913 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:48.834349 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:48.836555 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:48.868214 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:48.888103 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:48.897163 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:48.906504 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:48.934506 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:48.994169 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:49.025424 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.042881 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:49.064435 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:49.114627 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:49.119685 5034 raft_consensus.cc:482] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:49.119836 5034 raft_consensus.cc:504] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:49.120335 5034 leader_election.cc:288] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:49.122174 27194 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:49.122295 27193 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:49.122385 27193 leader_election.cc:302] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:49.122596 5034 raft_consensus.cc:2726] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:49.127313 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.135499 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:49.149184 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.151379 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:49.152170 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:49.176364 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.198518 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.207096 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.212141 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.217134 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.219220 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.223269 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:49.269675 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.274395 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.276575 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:18:49.281672 5077 raft_consensus.cc:482] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:49.281862 5077 raft_consensus.cc:504] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:49.282531 5077 leader_election.cc:288] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:49.283839 30357 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:49.286860 30358 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:49.286943 30358 leader_election.cc:302] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:49.287142 5077 raft_consensus.cc:2726] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:49.326419 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.345760 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.366604 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:18:49.402566 5034 raft_consensus.cc:482] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:49.402709 5034 raft_consensus.cc:504] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:49.403164 5034 leader_election.cc:288] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:49.404310 27191 leader_election.cc:334] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:49.404507 27194 leader_election.cc:334] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:49.404609 27194 leader_election.cc:302] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:49.404830 5034 raft_consensus.cc:2726] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:49.418720 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.475769 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.483390 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.485193 30096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 205 similar messages]
W20240502 08:18:49.485960 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.487879 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:49.488783 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:18:49.501057 4617 raft_consensus.cc:482] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:49.501199 4617 raft_consensus.cc:504] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:49.501682 4617 leader_election.cc:288] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:49.502920 29963 leader_election.cc:334] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:49.503168 29964 leader_election.cc:334] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:49.503266 29964 leader_election.cc:302] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:49.503582 4617 raft_consensus.cc:2726] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:49.504889 27191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 175 similar messages]
W20240502 08:18:49.519497 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.522683 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:49.532573 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.547133 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.559564 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.559748 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.568447 29963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240502 08:18:49.596469 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.607880 27587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240502 08:18:49.614267 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:49.620427 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.631775 30355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 08:18:49.641427 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.689184 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.706167 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.721519 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.758654 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.792152 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.842020 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:49.859632 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.899451 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.903903 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.979236 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.979561 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:49.985757 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:49.993467 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:50.013531 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.015266 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.058389 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.121367 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:18:50.162349 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.171343 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.179021 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:50.237699 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.253832 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.270253 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:18:50.276665 4827 raft_consensus.cc:482] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:50.276860 4827 raft_consensus.cc:504] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:50.277549 4827 leader_election.cc:288] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:50.279948 30095 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:50.280506 30096 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:50.280618 30096 leader_election.cc:302] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:50.281149 4827 raft_consensus.cc:2726] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:18:50.320333 4990 raft_consensus.cc:482] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:50.320457 4990 raft_consensus.cc:504] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:18:50.320909 4990 leader_election.cc:288] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:50.322342 27587 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:50.322643 27584 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:50.322772 27584 leader_election.cc:302] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:50.322965 4990 raft_consensus.cc:2726] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:50.395403 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.437292 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.439307 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:50.444482 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:50.449769 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.478906 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.528391 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:18:50.568133 4990 raft_consensus.cc:482] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:50.568303 4990 raft_consensus.cc:504] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:50.568992 4990 leader_election.cc:288] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:50.570096 27586 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:50.570279 27584 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:50.570374 27584 leader_election.cc:302] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:50.570585 4990 raft_consensus.cc:2726] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:50.578667 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.598665 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.666169 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.723106 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.726812 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.766942 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:50.782429 4617 raft_consensus.cc:482] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:50.782585 4617 raft_consensus.cc:504] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:50.783264 4617 leader_election.cc:288] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:50.786142 29964 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:50.788261 29963 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:50.788396 29963 leader_election.cc:302] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:50.788707 5098 raft_consensus.cc:2726] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:50.797894 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.839821 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.847472 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:50.857697 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:50.949491 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:50.974051 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:51.047049 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.048250 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:51.048249 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.078040 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:51.097939 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:51.209447 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.261535 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:51.338991 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.388603 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:18:51.433045 5269 raft_consensus.cc:482] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:51.433207 5269 raft_consensus.cc:504] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:18:51.433714 5269 leader_election.cc:288] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:51.434995 30355 leader_election.cc:334] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:51.435106 30357 leader_election.cc:334] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:51.435210 30357 leader_election.cc:302] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:51.435497 5269 raft_consensus.cc:2726] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:51.460062 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.481333 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.507809 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:51.518054 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:51.522353 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:51.586525 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:51.593739 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:51.594610 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.606988 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.701402 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:51.712741 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.713882 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:51.715425 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.733196 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.746178 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.751935 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.760167 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.781268 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.783080 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:51.786074 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.801925 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.819882 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.837577 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.839608 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.843559 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.868738 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:51.914219 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.917565 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.957783 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:51.979486 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.018224 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:52.020470 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.021831 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.033776 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.046591 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.057072 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.081269 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:52.097844 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.175936 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.189764 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.240119 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.260143 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:52.266808 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.273243 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.293511 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.296515 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.335426 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:52.336885 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.342466 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.375885 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.401002 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.418988 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:52.427877 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.435623 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.439713 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:18:52.479980 4990 raft_consensus.cc:482] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:52.480136 4990 raft_consensus.cc:504] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:52.480751 4990 leader_election.cc:288] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:52.481827 27586 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:52.482551 27587 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:52.482646 27587 leader_election.cc:302] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:52.482839 4990 raft_consensus.cc:2726] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:52.485340 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.517153 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:52.588059 4990 raft_consensus.cc:482] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:52.588202 4990 raft_consensus.cc:504] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:52.588735 4990 leader_election.cc:288] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:52.590049 27584 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:52.590181 27587 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:52.590297 27587 leader_election.cc:302] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:52.590494 4990 raft_consensus.cc:2726] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:52.620398 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.631510 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.643086 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.684455 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:18:52.728109 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.728586 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.742702 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:52.744980 4617 raft_consensus.cc:482] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:52.745152 4617 raft_consensus.cc:504] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:52.745734 4617 leader_election.cc:288] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:52.747318 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.747638 29964 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:52.748108 29961 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:52.748191 29961 leader_election.cc:302] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:52.748392 4617 raft_consensus.cc:2726] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:52.784744 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.811228 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.859238 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:52.892830 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.929970 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:52.951685 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:52.981556 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:53.053025 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:53.103540 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:53.173138 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:53.203305 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:18:53.235528 5379 raft_consensus.cc:482] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:53.235702 5379 raft_consensus.cc:504] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:53.236263 5379 leader_election.cc:288] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:53.238291 29961 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:53.239053 29964 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:53.239156 29964 leader_election.cc:302] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:53.239354 5379 raft_consensus.cc:2726] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:53.249295 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:53.272033 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:53.327198 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:53.362442 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:53.456440 4990 raft_consensus.cc:482] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:53.456616 4990 raft_consensus.cc:504] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:18:53.457217 4990 leader_election.cc:288] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:53.458985 27584 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:53.459116 27586 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:53.459228 27586 leader_election.cc:302] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:53.459440 4990 raft_consensus.cc:2726] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:53.462060 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:53.489431 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:53.491894 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:53.520888 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:53.548719 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:53.559935 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:53.577001 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:53.615613 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:53.658413 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:53.667174 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:53.697058 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:53.787777 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:53.807094 5267 raft_consensus.cc:482] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:53.807227 5267 raft_consensus.cc:504] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:18:53.807878 5267 leader_election.cc:288] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:53.809150 30095 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:53.809489 30096 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:53.809592 30096 leader_election.cc:302] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:53.809921 5267 raft_consensus.cc:2726] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:53.880528 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:53.985638 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:54.043551 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.051100 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.080678 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:54.084014 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:54.092269 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:18:54.093050 5604 raft_consensus.cc:482] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:54.093181 5604 raft_consensus.cc:504] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:54.093885 5604 leader_election.cc:288] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:54.095212 27586 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:54.095403 27587 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:54.095481 27587 leader_election.cc:302] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:54.095698 5604 raft_consensus.cc:2726] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:54.128652 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.138141 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.161303 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.170204 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.175014 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.195999 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.197422 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.206225 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:54.230516 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.256561 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:18:54.281172 5379 raft_consensus.cc:482] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:54.281311 5379 raft_consensus.cc:504] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:18:54.281868 5379 leader_election.cc:288] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:54.283063 29964 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:54.283181 29961 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:54.283278 29961 leader_election.cc:302] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:54.283514 5379 raft_consensus.cc:2726] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:54.310485 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.322588 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.326817 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:54.342897 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:54.346733 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.365350 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.368757 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:54.407964 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.428184 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.440740 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:54.449862 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.451589 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:54.457134 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:54.476364 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.502379 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.515081 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.520655 30096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 214 similar messages]
W20240502 08:18:54.521428 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.538888 27191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240502 08:18:54.573853 29961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240502 08:18:54.574571 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.597025 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.608173 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.625746 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.627044 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.639854 27586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 175 similar messages]
W20240502 08:18:54.647264 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:54.675263 30357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240502 08:18:54.682490 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.685628 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.706615 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.706929 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:54.707603 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:54.721261 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.747566 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.764825 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.808543 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:18:54.898730 5597 raft_consensus.cc:482] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:54.898870 5597 raft_consensus.cc:504] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:54.899451 5597 leader_election.cc:288] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:54.900604 29964 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:54.900743 29963 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:54.900835 29963 leader_election.cc:302] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:54.901003 5597 raft_consensus.cc:2726] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:54.905800 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.917166 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:54.948593 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:18:54.960263 5604 raft_consensus.cc:482] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:54.960417 5604 raft_consensus.cc:504] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:18:54.961019 5604 leader_election.cc:288] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:54.962242 27586 leader_election.cc:334] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:54.962375 27584 leader_election.cc:334] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:54.962472 27584 leader_election.cc:302] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:54.962648 5604 raft_consensus.cc:2726] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:54.973811 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:55.003664 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.046053 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:55.049686 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.064149 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.064858 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:18:55.098848 5597 raft_consensus.cc:482] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:18:55.099026 5597 raft_consensus.cc:504] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:18:55.099689 5597 leader_election.cc:288] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:55.100741 29963 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:55.100941 29964 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:55.101012 29964 leader_election.cc:302] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:55.101214 5597 raft_consensus.cc:2726] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:55.102387 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.105121 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.152096 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.195170 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:55.218436 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.222910 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.276557 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.281364 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.299691 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.349295 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:18:55.377322 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.460557 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.470052 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.482466 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:55.491312 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:55.556315 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.689524 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:55.716141 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:18:55.717892 5604 raft_consensus.cc:482] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:55.718060 5604 raft_consensus.cc:504] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:55.718688 5604 leader_election.cc:288] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:55.720191 27586 leader_election.cc:334] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:55.720355 27584 leader_election.cc:334] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:55.720443 27584 leader_election.cc:302] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:55.720654 5604 raft_consensus.cc:2726] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:55.734246 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.791287 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.901825 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.912660 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.915037 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:55.961239 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:55.977602 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:56.006618 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:56.015169 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.074294 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.086884 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:56.110553 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:56.121273 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:18:56.184432 5789 raft_consensus.cc:482] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:56.184569 5789 raft_consensus.cc:504] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:56.185078 5789 leader_election.cc:288] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:56.186203 27193 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:56.186326 27194 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:56.186429 27194 leader_election.cc:302] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:56.186733 5789 raft_consensus.cc:2726] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:56.196053 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:56.235836 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.273314 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:18:56.354169 4990 raft_consensus.cc:482] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:56.354350 4990 raft_consensus.cc:504] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:18:56.355032 4990 leader_election.cc:288] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:56.356948 27587 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:56.357281 27586 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:56.357424 27586 leader_election.cc:302] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:56.357725 4990 raft_consensus.cc:2726] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:56.360940 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:56.386665 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.459064 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.467826 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:56.510629 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:56.523131 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.545783 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:56.547911 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.549960 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:56.578791 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.592795 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.597550 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.615679 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.655951 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.656491 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:56.685139 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.711618 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.728435 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.745520 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.751506 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:56.760735 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.761150 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.774518 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.774502 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:56.782153 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.792142 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.828930 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.856534 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.870584 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.885522 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.925050 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:56.953019 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.971387 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:56.971920 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:56.984647 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:57.009801 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.020476 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.026508 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.029973 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.061993 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.062913 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.066358 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:57.087550 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.095494 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.120905 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.177101 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:57.221513 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.255831 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.283754 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:57.366886 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.382253 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:57.395848 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.399675 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.423498 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.425972 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:57.466423 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.473896 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.474016 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.548214 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:18:57.567692 5790 raft_consensus.cc:482] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:57.567831 5790 raft_consensus.cc:504] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:57.568361 5790 leader_election.cc:288] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:18:57.569775 27194 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:57.569918 27193 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:18:57.570011 27194 leader_election.cc:302] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:57.570235 5790 raft_consensus.cc:2726] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:18:57.582334 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.631706 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.634318 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:57.672217 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.732095 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:18:57.762825 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:57.775988 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.796635 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.797963 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.859195 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.859939 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.920279 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.957569 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:57.967264 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:58.025524 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:58.058070 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:58.150638 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:58.163141 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:58.182111 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:58.316874 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:58.347347 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:58.365345 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:58.417485 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:58.469650 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:58.542834 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:58.554955 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:18:58.611347 5706 raft_consensus.cc:482] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:58.611521 5706 raft_consensus.cc:504] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:18:58.612267 5706 leader_election.cc:288] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:58.614543 29963 leader_election.cc:334] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:58.614722 29964 leader_election.cc:334] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:58.614804 29964 leader_election.cc:302] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:58.615064 5706 raft_consensus.cc:2726] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:58.620357 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:58.643159 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:58.671458 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:58.681473 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:58.683449 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:58.695900 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:58.706317 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:58.784741 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:58.791461 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:58.864902 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:58.931363 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:58.948058 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:58.948019 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:58.972935 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:58.999207 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:59.055341 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.062021 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.064020 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.064410 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:59.065326 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.075992 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:59.135076 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.165342 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.203850 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.212917 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.231390 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:59.241992 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.263175 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.277702 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.288136 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.319259 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.319938 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.347685 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.348943 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.361759 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.375216 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.390770 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.394054 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.399241 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:59.450475 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.450659 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:59.468690 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:59.468886 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.485220 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.489701 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:59.495945 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:18:59.508976 5267 raft_consensus.cc:482] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:18:59.509127 5267 raft_consensus.cc:504] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:18:59.509716 5267 leader_election.cc:288] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:18:59.510928 30095 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:59.511126 30093 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:18:59.511217 30093 leader_election.cc:302] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:18:59.511451 5267 raft_consensus.cc:2726] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:18:59.512708 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:18:59.538952 5790 raft_consensus.cc:482] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:18:59.539134 5790 raft_consensus.cc:504] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:18:59.539745 5790 leader_election.cc:288] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:18:59.540613 27194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 174 similar messages]
W20240502 08:18:59.541673 27193 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:18:59.541823 27194 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:18:59.541915 27194 leader_election.cc:302] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:18:59.542318 5790 raft_consensus.cc:2726] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:18:59.557718 30093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 212 similar messages]
W20240502 08:18:59.559410 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.563058 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.567303 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.598937 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.612308 29963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240502 08:18:59.613175 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.642145 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.652014 27587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 166 similar messages]
W20240502 08:18:59.699322 30357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240502 08:18:59.700388 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.700498 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:18:59.751904 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.772583 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:18:59.826479 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.845265 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.909747 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.921855 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.929868 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:18:59.963706 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.978714 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:18:59.992164 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:00.006503 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:00.056752 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:00.128530 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:00.141340 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:19:00.150205 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:00.207405 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:00.219122 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:00.238610 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:19:00.257753 5604 raft_consensus.cc:482] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:00.257891 5604 raft_consensus.cc:504] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:19:00.258543 5604 leader_election.cc:288] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:00.259831 27584 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:00.260001 27586 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:00.260100 27586 leader_election.cc:302] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:00.260303 5604 raft_consensus.cc:2726] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:00.262624 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:00.311244 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:00.333570 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:00.420426 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:00.435289 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:00.438436 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:19:00.440716 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:00.445495 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:00.536417 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:00.537209 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:00.562119 6112 raft_consensus.cc:482] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:00.562283 6112 raft_consensus.cc:504] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:19:00.562870 6112 leader_election.cc:288] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:00.564527 29961 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:00.564637 29963 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:00.564713 29963 leader_election.cc:302] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:00.565057 6112 raft_consensus.cc:2726] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:00.565063 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:00.660811 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:00.698588 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:00.721782 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:00.830938 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:00.871677 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:00.884855 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:01.004921 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.012537 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:01.086585 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:01.118781 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:19:01.157806 5706 raft_consensus.cc:482] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:01.157956 5706 raft_consensus.cc:504] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:01.158587 5706 leader_election.cc:288] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:01.160153 29964 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:01.160326 29961 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:01.160434 29961 leader_election.cc:302] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:01.160641 5706 raft_consensus.cc:2726] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:01.266655 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:01.283481 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.288375 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:01.296097 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:01.297142 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.335462 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:01.344197 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:01.366621 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.432332 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.453552 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.467188 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:01.488814 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.499846 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:01.531140 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.543437 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.543517 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:01.552640 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.564608 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.582458 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:01.602268 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:01.612056 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.648952 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:01.695297 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.697808 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:01.700886 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.740878 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.774534 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:01.774672 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.774691 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:19:01.791278 6262 raft_consensus.cc:482] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:01.791450 6262 raft_consensus.cc:504] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:01.792250 6262 leader_election.cc:288] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:01.794662 27587 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:01.794878 27584 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:01.795001 27584 leader_election.cc:302] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:01.795214 6262 raft_consensus.cc:2726] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:01.831544 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.879635 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.889449 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:01.899065 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.911653 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.914700 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:01.920413 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.941198 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.945211 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:19:01.945739 6262 raft_consensus.cc:482] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:01.945856 6262 raft_consensus.cc:504] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:19:01.946476 6262 leader_election.cc:288] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:01.947593 27586 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:01.947758 27584 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:01.947835 27584 leader_election.cc:302] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:01.948040 6262 raft_consensus.cc:2726] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:01.957532 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:01.974272 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.006384 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.031448 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:02.035198 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.040791 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:02.047451 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.085991 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.090178 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.095499 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.108804 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.124383 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.145910 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.147320 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.189854 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.213088 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.283265 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.294839 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.314946 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:02.336036 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:02.377543 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.387005 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.452322 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.470485 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:02.491746 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.531401 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:19:02.578887 5706 raft_consensus.cc:482] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:02.579053 5706 raft_consensus.cc:504] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:19:02.579555 5706 leader_election.cc:288] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:02.580713 29961 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:02.580830 29963 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:02.580932 29963 leader_election.cc:302] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:02.581099 5706 raft_consensus.cc:2726] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:02.586786 27103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:02.587754 27103 catalog_manager.cc:4573] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6cf739b314d84db08bc4cb32b0e5b0f4: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:02.631407 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:02.649377 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.659814 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.672303 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.723737 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.729511 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:02.739385 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.772405 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.825752 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.886941 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.906826 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.922995 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:19:02.953526 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.970196 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.976260 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:02.991760 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:03.004807 6175 raft_consensus.cc:482] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:03.004972 6175 raft_consensus.cc:504] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:03.005707 6175 leader_election.cc:288] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:03.007336 27191 leader_election.cc:334] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:03.007680 27194 leader_election.cc:334] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:03.007807 27194 leader_election.cc:302] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:03.008016 6175 raft_consensus.cc:2726] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:03.010315 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:03.097824 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:03.141645 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:03.147608 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:03.169090 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:03.247259 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:03.253420 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:03.407358 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:03.433172 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:03.514943 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:03.548651 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:03.552738 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:03.573364 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:03.584424 5267 raft_consensus.cc:482] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:03.584558 5267 raft_consensus.cc:504] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:03.585063 5267 leader_election.cc:288] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:03.586110 30093 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:03.586231 30095 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:03.586339 30095 leader_election.cc:302] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:03.586551 5267 raft_consensus.cc:2726] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:19:03.647163 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:03.720109 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:03.726366 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:03.788993 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:03.792991 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:03.830817 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:03.863462 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:03.877537 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:03.914302 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:03.932536 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:03.937038 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:03.988648 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:04.003813 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.019918 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:04.027177 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.084280 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:04.084412 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.108047 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.119227 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.131136 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.166329 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.173205 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.187223 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:04.210332 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:04.216181 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.243130 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.244983 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.259052 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.265655 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.268695 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.269479 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:04.281378 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:04.290151 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:04.314183 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:04.330701 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.333199 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.369685 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.371318 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.384243 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.400059 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.439046 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.440455 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.440889 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:04.463168 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.471791 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.477349 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.533491 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.547483 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.566771 27191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 168 similar messages]
W20240502 08:19:04.571345 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.577148 30093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 215 similar messages]
W20240502 08:19:04.595798 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.603933 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:04.611658 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.621768 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.637621 29964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240502 08:19:04.656332 27587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 166 similar messages]
W20240502 08:19:04.665782 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.675079 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:04.722934 30357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240502 08:19:04.783867 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:04.813338 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:04.824831 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:04.840746 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.854466 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.878847 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:04.955106 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:05.024758 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:05.026021 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:19:05.030107 6458 raft_consensus.cc:482] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:05.030252 6458 raft_consensus.cc:504] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:19:05.030782 6458 leader_election.cc:288] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:05.031786 27191 leader_election.cc:334] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:05.031930 27193 leader_election.cc:334] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:05.032022 27193 leader_election.cc:302] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:05.032213 6458 raft_consensus.cc:2726] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:05.044855 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:05.102967 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:05.105125 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:05.122435 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:05.142771 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:19:05.147205 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:05.213562 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:05.241608 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:05.320739 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:05.371747 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:05.375849 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:05.406896 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:05.413748 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:05.520712 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:05.536367 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:05.578189 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:05.611109 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:05.631397 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:05.684540 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:05.721583 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:05.723500 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:05.728494 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:19:05.742379 6326 raft_consensus.cc:482] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:05.742544 6326 raft_consensus.cc:504] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:19:05.743227 6326 leader_election.cc:288] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:05.744495 27584 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:05.744876 27587 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:05.745008 27587 leader_election.cc:302] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:05.745218 6326 raft_consensus.cc:2726] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:05.745520 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:05.785928 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:05.802515 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:05.974469 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:06.068079 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:06.105867 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:06.172828 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:06.179811 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.186900 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:06.217149 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:06.223230 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.261556 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.289963 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.313261 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:19:06.318332 6693 raft_consensus.cc:482] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:19:06.318457 6693 raft_consensus.cc:504] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:19:06.318966 6693 leader_election.cc:288] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:06.320133 27193 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:06.320742 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:06.320900 27194 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:06.320982 27194 leader_election.cc:302] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:06.321153 6693 raft_consensus.cc:2726] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:19:06.330699 6499 raft_consensus.cc:482] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:06.330827 6499 raft_consensus.cc:504] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:19:06.331476 6499 leader_election.cc:288] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:06.332322 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.332576 29964 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:06.332803 29961 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:06.332908 29961 leader_election.cc:302] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:06.333133 6499 raft_consensus.cc:2726] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:06.366055 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:06.376189 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:06.378486 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:06.404995 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:06.484526 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.497319 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.511370 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.581594 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.584935 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.589954 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.604590 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.626385 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:19:06.629132 5378 raft_consensus.cc:482] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:06.629264 5378 raft_consensus.cc:504] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:06.629802 5378 leader_election.cc:288] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:06.631172 30095 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:06.631314 30093 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:06.631397 30093 leader_election.cc:302] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:06.631577 5378 raft_consensus.cc:2726] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:19:06.660473 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:06.676920 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.683081 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:06.730291 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.741240 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.753013 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.755733 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.792551 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.793821 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.805222 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:06.805429 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.811584 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.814599 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:06.826409 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:06.836512 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.891469 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.901791 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:06.925201 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:06.987521 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.015534 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.049207 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.058665 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.086565 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:07.102370 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.116750 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:07.129803 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.135115 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.141674 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.149909 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.180564 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:07.214843 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.227008 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:07.238867 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:07.280267 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.291991 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.299932 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.330087 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.339731 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.342837 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:07.385185 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.391376 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.395996 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.415829 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.439288 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.440559 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.492173 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.561148 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:19:07.666947 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.696470 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:07.703289 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.766449 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.769675 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:07.776160 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:07.853619 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.888492 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.927363 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:07.945909 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:08.015784 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:08.045161 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:08.103871 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:08.136011 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:08.164232 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:08.197827 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:08.251684 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:08.293713 6715 raft_consensus.cc:482] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:08.293860 6715 raft_consensus.cc:504] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:19:08.294416 6715 leader_election.cc:288] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:08.295887 30355 leader_election.cc:334] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:08.296000 30357 leader_election.cc:334] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:08.296083 30357 leader_election.cc:302] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:08.296293 6715 raft_consensus.cc:2726] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:08.311995 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:08.323335 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:08.348335 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:19:08.401688 6715 raft_consensus.cc:482] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:19:08.401847 6715 raft_consensus.cc:504] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:19:08.402454 6715 leader_election.cc:288] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:08.404222 30357 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:08.404341 30358 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:08.404518 30358 leader_election.cc:302] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:08.404774 6715 raft_consensus.cc:2726] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:08.430323 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:08.446848 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:08.447695 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:08.571278 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:08.575768 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:19:08.604918 6499 raft_consensus.cc:482] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:08.605053 6499 raft_consensus.cc:504] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:19:08.605598 6499 leader_election.cc:288] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:08.606913 29963 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:08.607028 29964 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:08.607065 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:08.607107 29964 leader_election.cc:302] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:08.607318 6499 raft_consensus.cc:2726] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:08.639449 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:08.683029 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:08.732009 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:08.738937 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:08.747300 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:08.786921 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:08.793648 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:08.805202 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:08.836324 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:08.845708 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:08.925783 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:08.949193 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:08.957340 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:08.957564 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:08.958333 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:08.970770 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:08.979103 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.063715 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:09.081269 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.108273 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.136920 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:09.145978 6638 raft_consensus.cc:482] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:09.146147 6638 raft_consensus.cc:504] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:19:09.146728 6638 leader_election.cc:288] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:09.148008 27586 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:09.148636 27584 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:09.148734 27584 leader_election.cc:302] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:09.149046 6638 raft_consensus.cc:2726] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:09.165125 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:09.172609 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.181857 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.211138 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.216404 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.228147 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:09.248078 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.248802 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.283407 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:09.285785 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.288209 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.288444 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:19:09.328622 6326 raft_consensus.cc:482] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:09.328791 6326 raft_consensus.cc:504] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:09.329319 6326 leader_election.cc:288] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:09.330516 27587 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:09.330914 27584 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:09.330996 27584 leader_election.cc:302] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:09.331183 6326 raft_consensus.cc:2726] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:09.333498 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.348891 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.350894 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:09.357024 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.364010 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.368134 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.418576 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:09.427477 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:09.443673 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.493896 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:09.512676 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.533053 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:09.552781 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.559521 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.561692 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.565696 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:09.610625 27193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 175 similar messages]
W20240502 08:19:09.630631 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:19:09.631457 6950 raft_consensus.cc:482] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:09.631619 6950 raft_consensus.cc:504] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:19:09.632158 6950 leader_election.cc:288] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:09.633625 29961 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:09.634416 29964 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:09.634572 29964 leader_election.cc:302] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:09.635074 6950 raft_consensus.cc:2726] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:09.661706 29963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240502 08:19:09.664053 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.667042 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:09.689745 27584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 171 similar messages]
W20240502 08:19:09.701705 30093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 216 similar messages]
W20240502 08:19:09.711766 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.716315 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.728456 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.734597 30358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240502 08:19:09.747067 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:19:09.761324 6638 raft_consensus.cc:482] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:19:09.761471 6638 raft_consensus.cc:504] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:19:09.762148 6638 leader_election.cc:288] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:09.763221 27586 leader_election.cc:334] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:09.763368 27584 leader_election.cc:334] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:09.763450 27584 leader_election.cc:302] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:09.763660 6638 raft_consensus.cc:2726] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:09.831197 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.884166 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.908838 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.912843 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.921876 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.956535 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:09.973748 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:09.997555 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:10.009825 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:10.052945 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:10.087249 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:10.103559 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:19:10.132889 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:10.142256 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:10.150290 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:10.228801 6638 raft_consensus.cc:482] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:10.228950 6638 raft_consensus.cc:504] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:19:10.229434 6638 leader_election.cc:288] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:10.231165 27587 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:10.231359 27584 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:10.231456 27584 leader_election.cc:302] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:10.231704 6638 raft_consensus.cc:2726] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:10.248129 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:10.297905 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:10.308996 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:10.353765 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:10.363826 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:10.380184 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:10.388952 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:10.392935 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:10.459573 6950 raft_consensus.cc:482] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:10.459741 6950 raft_consensus.cc:504] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:19:10.460332 6950 leader_election.cc:288] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:10.461454 29964 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:10.461565 29963 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:10.461653 29964 leader_election.cc:302] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:10.461871 6950 raft_consensus.cc:2726] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:10.517902 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:10.531292 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:10.543401 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:19:10.549845 6499 raft_consensus.cc:482] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:10.549974 6499 raft_consensus.cc:504] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:10.550642 6499 leader_election.cc:288] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:10.551765 29964 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:10.552083 29961 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:10.552170 29961 leader_election.cc:302] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:10.552371 6499 raft_consensus.cc:2726] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:10.605037 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:10.642828 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:10.750959 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:10.803483 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:10.831089 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:10.907861 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:10.931924 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:10.942914 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:10.950651 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:10.967890 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:19:10.977658 6638 raft_consensus.cc:482] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:10.977788 6638 raft_consensus.cc:504] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:10.978296 6638 leader_election.cc:288] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:10.979473 27587 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:10.979588 27584 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:10.979673 27584 leader_election.cc:302] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:10.979846 6638 raft_consensus.cc:2726] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:11.085923 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:11.104382 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:11.107714 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.116060 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:11.125453 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:11.147907 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:11.170845 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.197311 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.241691 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:11.289630 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:11.346856 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.369366 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.385345 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:11.385764 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:11.401687 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:19:11.425189 6499 raft_consensus.cc:482] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:11.425345 6499 raft_consensus.cc:504] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:19:11.425933 6499 leader_election.cc:288] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:11.427281 29961 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:11.427517 29964 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:11.427613 29964 leader_election.cc:302] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:11.427803 6499 raft_consensus.cc:2726] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:11.438150 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:11.469463 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.500983 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:19:11.504668 6499 raft_consensus.cc:482] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:19:11.504820 6499 raft_consensus.cc:504] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:19:11.505329 6499 leader_election.cc:288] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:11.506426 29963 leader_election.cc:334] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:11.506582 29964 leader_election.cc:334] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:11.506695 29964 leader_election.cc:302] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:11.506897 6499 raft_consensus.cc:2726] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:11.512782 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.517964 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.560142 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.572228 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.583038 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.596977 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.645925 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.659601 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.662923 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.684621 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:11.686025 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:11.697147 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.723574 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.726168 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:11.731370 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:11.746214 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.747692 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.749109 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.758987 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.830484 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.837021 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.850988 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.917342 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:11.940521 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:11.942812 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:11.957713 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:11.977833 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:19:11.999456 7186 raft_consensus.cc:482] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:11.999600 7186 raft_consensus.cc:504] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:19:12.000183 7186 leader_election.cc:288] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:12.001248 27586 leader_election.cc:334] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:12.001401 27584 leader_election.cc:334] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:12.001477 27584 leader_election.cc:302] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:12.001624 7186 raft_consensus.cc:2726] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:12.029567 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:12.040007 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.061744 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.069773 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.140836 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:12.144662 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.169050 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:12.179008 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.210209 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.215114 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.222544 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:12.225795 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.250316 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.287781 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.303136 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:19:12.309919 7179 raft_consensus.cc:482] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:12.310062 7179 raft_consensus.cc:504] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:19:12.310531 7179 leader_election.cc:288] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:12.311570 30095 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:12.312145 30096 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:12.312225 30096 leader_election.cc:302] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:12.312393 6952 raft_consensus.cc:2726] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:12.362298 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:12.430019 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.433472 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.475709 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.494781 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.509488 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.564852 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.662349 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:12.762250 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.766458 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:12.768083 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.776172 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.778812 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:12.829597 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:19:12.833281 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.848285 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.950603 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:12.996316 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:13.035714 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:19:13.078464 6499 raft_consensus.cc:482] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:13.078617 6499 raft_consensus.cc:504] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:19:13.079268 6499 leader_election.cc:288] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:13.080999 29963 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:13.081856 29964 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:13.081969 29964 leader_election.cc:302] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:13.082240 6499 raft_consensus.cc:2726] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:13.112412 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:13.119835 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:13.146531 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:13.201499 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:13.244508 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:13.252074 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:13.263199 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:13.263259 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:13.279850 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:13.337747 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:13.383224 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:13.383504 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:13.388988 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:13.391856 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:13.432752 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:13.439585 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:19:13.460901 7313 raft_consensus.cc:482] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:13.461035 7313 raft_consensus.cc:504] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:13.461594 7313 leader_election.cc:288] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:13.462539 27587 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:13.462692 27584 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:13.462803 27584 leader_election.cc:302] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:13.462985 7313 raft_consensus.cc:2726] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:13.492740 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:13.529616 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:13.546365 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:13.572999 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:19:13.679251 6638 raft_consensus.cc:482] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:13.679383 6638 raft_consensus.cc:504] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:19:13.679884 6638 leader_election.cc:288] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:13.681160 27586 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:13.681299 27584 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:13.681423 27584 leader_election.cc:302] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:13.681643 6638 raft_consensus.cc:2726] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:13.718686 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:13.719534 6638 raft_consensus.cc:482] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:19:13.719707 6638 raft_consensus.cc:504] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:19:13.720333 6638 leader_election.cc:288] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:13.723524 27586 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:13.723738 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:13.724421 27587 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:13.724550 27587 leader_election.cc:302] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:13.724892 6638 raft_consensus.cc:2726] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:13.795557 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:13.850677 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:13.850719 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:13.886209 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:13.892699 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:13.940631 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:13.992208 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:13.995157 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.018234 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.020783 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:14.045307 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.059979 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.060617 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.072499 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:14.127319 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.170428 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:14.179275 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:14.188664 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.219172 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:14.223304 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.234400 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.282371 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.302292 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.309700 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:14.320953 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:14.336308 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.337220 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:19:14.349129 7058 raft_consensus.cc:482] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:19:14.349277 7058 raft_consensus.cc:504] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:19:14.349866 7058 leader_election.cc:288] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:14.351177 30357 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:14.351610 30358 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:14.351701 30358 leader_election.cc:302] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:14.351887 7058 raft_consensus.cc:2726] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:14.360596 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:14.369532 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.404187 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.418689 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:14.432624 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:14.443749 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.452152 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:14.475474 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.476042 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.478628 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.518770 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.521165 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:19:14.526437 7014 raft_consensus.cc:482] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:19:14.526593 7014 raft_consensus.cc:504] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:19:14.527231 7014 leader_election.cc:288] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:14.528681 27193 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:14.529310 27194 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:14.529439 27194 leader_election.cc:302] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:14.529660 7014 raft_consensus.cc:2726] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:14.565712 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:14.570420 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.574707 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:14.615849 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:14.634769 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.654814 27194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 176 similar messages]
W20240502 08:19:14.655613 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.715781 30093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 212 similar messages]
W20240502 08:19:14.722509 27587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 172 similar messages]
W20240502 08:19:14.723313 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.735828 29961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240502 08:19:14.747901 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.755008 30355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 08:19:14.778216 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.788923 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.793037 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.798671 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.830968 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.864535 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:14.886546 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.894845 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.946419 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:14.973783 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:15.034210 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:15.084298 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:15.171113 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:15.185701 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:15.242045 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:15.260964 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:15.298578 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:15.304307 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:15.311010 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:15.334843 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:15.361074 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:19:15.363099 27105 maintenance_manager.cc:419] P c086547273ea42f0a31343f9f60d13f0: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033372
I20240502 08:19:15.419835 27104 maintenance_manager.cc:643] P c086547273ea42f0a31343f9f60d13f0: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.056s user 0.046s sys 0.008s Metrics: {"bytes_written":91243,"cfile_init":1,"dirs.queue_time_us":359,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1183,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1182,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":251,"thread_start_us":468,"threads_started":2,"wal-append.queue_time_us":297}
I20240502 08:19:15.420877 27105 maintenance_manager.cc:419] P c086547273ea42f0a31343f9f60d13f0: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 39769 bytes on disk
I20240502 08:19:15.421525 27104 maintenance_manager.cc:643] P c086547273ea42f0a31343f9f60d13f0: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240502 08:19:15.458062 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:15.520548 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:15.540746 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:15.564319 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:15.571940 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:15.577028 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:15.597203 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:15.679910 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:15.724853 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:15.740743 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:15.761127 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:19:15.761185 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:15.825318 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:15.870256 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:19:15.902552 7443 raft_consensus.cc:482] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:15.902729 7443 raft_consensus.cc:504] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:19:15.903422 7443 leader_election.cc:288] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:15.904752 30357 leader_election.cc:334] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:15.904886 30355 leader_election.cc:334] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:15.904978 30355 leader_election.cc:302] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:15.905135 7443 raft_consensus.cc:2726] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:15.922797 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:15.925179 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:15.934253 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:15.936781 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:15.944216 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:15.973359 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:19:15.986449 7399 raft_consensus.cc:482] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:15.986617 7399 raft_consensus.cc:504] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:15.987247 7399 leader_election.cc:288] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:15.989073 27584 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:15.989519 27587 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:15.989620 27587 leader_election.cc:302] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:15.989938 7399 raft_consensus.cc:2726] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:16.013788 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:16.044307 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:16.047158 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:16.070410 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:16.112406 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.121846 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:16.186211 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.278476 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.311892 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:16.359440 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.403791 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.413653 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:16.453961 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.483078 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.503468 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.506031 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.516527 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:16.522837 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:16.533207 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.597169 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:16.629982 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.631035 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.661566 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.664115 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.699275 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.728235 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:16.741864 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:16.767113 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.799896 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:16.802794 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.814793 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:16.837785 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:16.844972 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.897502 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.904701 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.911082 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.930315 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.933580 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.951686 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.953533 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:16.959844 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:16.965938 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:16.973788 7570 raft_consensus.cc:482] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:16.973948 7570 raft_consensus.cc:504] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:19:16.974781 7570 leader_election.cc:288] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:16.976094 27194 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:16.976238 27193 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:16.976342 27193 leader_election.cc:302] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:16.976596 7570 raft_consensus.cc:2726] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:19:16.979290 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.007115 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:17.020570 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:17.046072 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.056598 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.105021 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.151135 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:19:17.174513 7422 raft_consensus.cc:482] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:17.174690 7422 raft_consensus.cc:504] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:19:17.175379 7422 leader_election.cc:288] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:17.177511 30096 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:17.178108 30095 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:17.178223 30095 leader_election.cc:302] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:17.178534 7422 raft_consensus.cc:2726] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:17.178934 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.203536 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.224469 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.225795 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:17.252223 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:17.269158 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.296296 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.349102 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.402197 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.404762 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.406765 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:17.417073 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.493912 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.495083 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:19:17.509305 7570 raft_consensus.cc:482] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:17.509464 7570 raft_consensus.cc:504] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:19:17.510079 7570 leader_election.cc:288] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:17.511212 27191 leader_election.cc:334] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:17.511632 27193 leader_election.cc:334] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:17.511715 27193 leader_election.cc:302] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:17.511942 7685 raft_consensus.cc:2726] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:17.520874 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.524845 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:17.618604 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.700508 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.726662 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:17.784092 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.844405 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.866514 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.868813 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:17.876137 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.901469 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.907327 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.937065 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:17.945446 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:19:17.970799 7642 raft_consensus.cc:482] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:17.970932 7642 raft_consensus.cc:504] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:17.971550 7642 leader_election.cc:288] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:17.973043 29964 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:17.973780 29961 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:17.973891 29961 leader_election.cc:302] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:17.974275 7642 raft_consensus.cc:2726] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:17.987016 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:17.988803 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:19:17.995375 7576 raft_consensus.cc:482] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:19:17.995515 7576 raft_consensus.cc:504] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:19:17.996098 7576 leader_election.cc:288] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:17.997555 27586 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:17.997946 27587 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:17.998085 27587 leader_election.cc:302] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:17.998308 7576 raft_consensus.cc:2726] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:18.006795 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:18.066392 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:18.074319 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:18.180872 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:18.211583 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:18.276199 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:18.304061 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:19:18.360330 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:18.378508 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:18.378618 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:18.434420 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:18.447376 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:18.450942 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:18.452903 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:18.460405 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:18.518894 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:18.523031 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:18.578758 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:18.597604 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:18.619660 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:18.716917 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:18.723793 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:18.760394 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:18.789413 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:18.862288 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:18.863010 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:18.901326 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:18.904855 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:18.920320 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:18.932122 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:18.974403 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:18.990703 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.023687 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:19.072360 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.077450 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.103510 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:19.111469 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:19.128822 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.133545 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:19.159999 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.161401 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.170248 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:19.194576 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.196288 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.208221 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.263815 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.323195 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.342785 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.349272 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.363071 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:19.404891 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:19.455111 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.456159 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.465216 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.486959 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:19.492978 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.493804 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:19.500468 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.563853 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.588331 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:19.607002 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:19.607183 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.635589 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.647094 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.668105 27194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 177 similar messages]
W20240502 08:19:19.673591 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.692662 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.692745 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.705663 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.728971 30095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 215 similar messages]
W20240502 08:19:19.738644 29963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240502 08:19:19.742899 27584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 165 similar messages]
W20240502 08:19:19.763942 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:19.790691 30355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 08:19:19.844451 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.867108 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:19.899129 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:19:19.930159 7577 raft_consensus.cc:482] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:19.930310 7577 raft_consensus.cc:504] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:19:19.931033 7577 leader_election.cc:288] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:19.932113 29961 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:19.932410 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:19.932834 29963 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:19.932937 29963 leader_election.cc:302] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:19.933151 7577 raft_consensus.cc:2726] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:19.952847 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.966025 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.968339 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:19.973784 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:19.990757 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:20.015300 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:20.042843 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:20.054976 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:19:20.079515 7422 raft_consensus.cc:482] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:20.079658 7422 raft_consensus.cc:504] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:19:20.080163 7422 leader_election.cc:288] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:20.081732 30095 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:20.082069 30096 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:20.082165 30096 leader_election.cc:302] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:20.082370 7422 raft_consensus.cc:2726] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:20.135861 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:20.167798 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:19:20.174844 7576 raft_consensus.cc:482] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:20.174970 7576 raft_consensus.cc:504] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:19:20.175467 7576 leader_election.cc:288] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:20.176963 27586 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:20.177071 27584 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:20.177153 27584 leader_election.cc:302] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:20.177356 7576 raft_consensus.cc:2726] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:20.200845 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:20.227448 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:20.260953 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:20.279120 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:20.283582 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:20.320516 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:20.354404 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:20.412811 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:20.417146 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:20.433544 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:20.438966 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:20.462257 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:20.481470 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:20.588241 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:20.675426 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:20.676553 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:20.748129 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:20.762861 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:19:20.797659 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:20.832419 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:20.843302 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:20.926497 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:20.956270 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:20.962530 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:21.002681 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:21.008981 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:21.043963 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.054517 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:21.073031 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:21.079408 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.106988 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:21.136476 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:21.172897 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:21.243690 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.271539 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.322525 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.390720 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.421861 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:21.429615 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.463629 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:21.482053 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.505021 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:19:21.513715 7577 raft_consensus.cc:482] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:21.513876 7577 raft_consensus.cc:504] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:19:21.514436 7577 leader_election.cc:288] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:21.515565 29963 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:21.515743 29964 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:21.515867 29964 leader_election.cc:302] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:21.516070 7577 raft_consensus.cc:2726] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:21.531137 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.550755 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.557801 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:21.579278 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.603986 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.619758 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:21.636013 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:19:21.648658 7813 raft_consensus.cc:482] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:21.648782 7813 raft_consensus.cc:504] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:21.649296 7813 leader_election.cc:288] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:21.650514 27191 leader_election.cc:334] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:21.650732 27194 leader_election.cc:334] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:21.650846 27194 leader_election.cc:302] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:21.651069 7813 raft_consensus.cc:2726] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:21.685299 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:21.691476 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.714866 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:21.732992 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:21.740123 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.745242 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.776657 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.802562 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.823791 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.849040 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.867621 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:21.921023 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.942674 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.954417 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.963223 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.977142 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:21.988988 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.008018 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:22.009827 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:22.015156 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.042950 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.065860 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:22.096886 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.115092 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:22.189375 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:19:22.198660 8071 raft_consensus.cc:482] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:22.198812 8071 raft_consensus.cc:504] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:22.199466 8071 leader_election.cc:288] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:22.200912 27587 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:22.201021 27584 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:22.201139 27584 leader_election.cc:302] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:22.201393 8071 raft_consensus.cc:2726] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:22.204288 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:22.205631 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.224414 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:22.224555 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.225240 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.236217 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.256162 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.290182 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:19:22.334125 7577 raft_consensus.cc:482] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:22.334271 7577 raft_consensus.cc:504] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:19:22.334992 7577 leader_election.cc:288] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:22.336374 29963 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:22.337378 29961 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:22.337482 29961 leader_election.cc:302] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:22.337698 7577 raft_consensus.cc:2726] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:22.403079 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.429891 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.440083 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:22.455612 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.485392 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.513726 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.516845 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:22.525296 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.527267 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:19:22.574141 8115 raft_consensus.cc:482] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:22.574316 8115 raft_consensus.cc:504] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:19:22.574937 8115 leader_election.cc:288] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:22.576184 29963 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:22.576303 29964 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:22.576416 29964 leader_election.cc:302] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:22.576603 8115 raft_consensus.cc:2726] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:22.619282 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.651352 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.651472 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:22.655930 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.698647 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:19:22.725700 8016 raft_consensus.cc:482] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:22.725880 8016 raft_consensus.cc:504] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:19:22.726483 8016 leader_election.cc:288] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:22.727810 27587 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:22.727914 27584 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:22.727993 27584 leader_election.cc:302] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:22.728371 8071 raft_consensus.cc:2726] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:22.756610 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.763183 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:22.855791 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:22.946223 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.955266 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:22.958658 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:23.010341 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:23.022573 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:23.035969 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:23.063023 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:23.069830 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:23.094168 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:23.124861 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:23.127024 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:23.138196 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:23.174841 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:23.174925 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:19:23.233862 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:23.252614 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:23.311436 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:23.345497 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:23.393590 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:23.415678 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:23.468966 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:23.490741 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:23.513525 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:23.543900 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:23.584528 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:23.605147 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:23.671818 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:23.696962 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:23.735729 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:23.786134 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:23.917934 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:23.918179 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:23.921290 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:23.922111 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:23.946652 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:24.051890 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.054597 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:24.066617 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.072904 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:24.090989 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.112776 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:24.118723 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.120136 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.120221 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.121811 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.162029 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.179244 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:24.228626 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:24.236233 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.239616 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.300957 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.324275 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.373339 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:24.434453 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.443991 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.444398 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.449890 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.452327 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.507090 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:24.528157 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.550069 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.572222 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.582109 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.596203 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:24.597201 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.649384 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.653550 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.706857 27191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 174 similar messages]
W20240502 08:19:24.741273 30093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 211 similar messages]
W20240502 08:19:24.741986 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:24.756522 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.763856 29963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240502 08:19:24.773231 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.776855 27584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 173 similar messages]
W20240502 08:19:24.783645 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.792531 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:24.813166 30357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240502 08:19:24.814288 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:24.850370 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.855971 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.886904 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.899248 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:24.925190 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.931658 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.932314 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:24.932710 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:24.979733 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:24.993980 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:25.047698 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:25.127530 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:25.149298 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:19:25.166070 8316 raft_consensus.cc:482] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:25.166208 8316 raft_consensus.cc:504] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:25.166819 8316 leader_election.cc:288] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:25.168102 30095 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:25.168622 30093 leader_election.cc:334] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:25.168727 30093 leader_election.cc:302] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:25.168943 8316 raft_consensus.cc:2726] T a793eaaeba89458baf8af7caf3f24c70 P 84cda20738d2452da166e31fc6da57a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:19:25.188570 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:19:25.192739 7577 raft_consensus.cc:482] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:25.192859 7577 raft_consensus.cc:504] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:19:25.193320 7577 leader_election.cc:288] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:25.194557 29964 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:25.194927 29961 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:25.195008 29961 leader_election.cc:302] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:25.195187 7577 raft_consensus.cc:2726] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:25.221645 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:25.245453 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:25.282058 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:25.286724 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:19:25.299974 27698 maintenance_manager.cc:419] P bf8ac71b15f740a0a928911360773442: Scheduling FlushMRSOp(15a333c65455473fbdc26d1ec137aebc): perf score=0.033337
W20240502 08:19:25.306887 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:25.338099 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:19:25.373585 27588 maintenance_manager.cc:643] P bf8ac71b15f740a0a928911360773442: FlushMRSOp(15a333c65455473fbdc26d1ec137aebc) complete. Timing: real 0.073s user 0.058s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":390,"dirs.queue_time_us":625,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1698,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":693,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1261,"threads_started":3,"wal-append.queue_time_us":397}
W20240502 08:19:25.376085 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:19:25.376343 27698 maintenance_manager.cc:419] P bf8ac71b15f740a0a928911360773442: Scheduling UndoDeltaBlockGCOp(15a333c65455473fbdc26d1ec137aebc): 4586 bytes on disk
I20240502 08:19:25.377064 27588 maintenance_manager.cc:643] P bf8ac71b15f740a0a928911360773442: UndoDeltaBlockGCOp(15a333c65455473fbdc26d1ec137aebc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 08:19:25.392282 27304 maintenance_manager.cc:419] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Scheduling FlushMRSOp(af2f11471c064f48924191fc2607d31c): perf score=0.033368
W20240502 08:19:25.470615 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:25.502575 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:25.533339 8322 raft_consensus.cc:482] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:19:25.533499 8322 raft_consensus.cc:504] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:19:25.533342 27195 maintenance_manager.cc:643] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: FlushMRSOp(af2f11471c064f48924191fc2607d31c) complete. Timing: real 0.140s user 0.091s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":5337,"dirs.queue_time_us":6382,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":4161,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":835,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":18252,"threads_started":3,"wal-append.queue_time_us":6387}
I20240502 08:19:25.534158 8322 leader_election.cc:288] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
I20240502 08:19:25.537043 27304 maintenance_manager.cc:419] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Scheduling UndoDeltaBlockGCOp(af2f11471c064f48924191fc2607d31c): 4926 bytes on disk
W20240502 08:19:25.538959 27586 leader_election.cc:334] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:25.539259 27584 leader_election.cc:334] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:25.539379 27584 leader_election.cc:302] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
W20240502 08:19:25.539903 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:19:25.540163 8322 raft_consensus.cc:2726] T 119ad5932c314235b85177cceeb2277c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:19:25.541011 27195 maintenance_manager.cc:643] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: UndoDeltaBlockGCOp(af2f11471c064f48924191fc2607d31c) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
W20240502 08:19:25.550869 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:25.582144 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:25.620003 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:25.621357 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:25.632366 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:25.645059 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:25.661628 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:19:25.687633 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:25.703737 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:25.719161 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:25.734239 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:25.774446 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:25.791148 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:25.839005 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:25.953138 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:25.982676 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:26.020910 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.043931 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.047644 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:26.098591 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:26.154718 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:19:26.183128 8322 raft_consensus.cc:482] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:26.183266 8322 raft_consensus.cc:504] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:26.183872 8322 leader_election.cc:288] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:26.185159 27587 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:26.185288 27584 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:26.185379 27584 leader_election.cc:302] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:26.185567 8322 raft_consensus.cc:2726] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:26.203186 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:26.274329 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:26.317807 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.319492 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:26.324465 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.351927 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.353395 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.359920 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.406945 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.425415 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.434123 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.444096 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.456847 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:19:26.483256 7577 raft_consensus.cc:482] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:26.483400 7577 raft_consensus.cc:504] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:26.483870 7577 leader_election.cc:288] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:26.485199 29961 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:26.485381 29964 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:26.485478 29964 leader_election.cc:302] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:26.485718 7577 raft_consensus.cc:2726] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:26.501483 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.502074 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:26.507033 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.537837 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.571460 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:26.621896 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.686193 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.686599 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.703733 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:26.769663 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:26.792438 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.798450 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.842238 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.874990 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:26.876375 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:19:26.924106 8426 raft_consensus.cc:482] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:19:26.924252 8426 raft_consensus.cc:504] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:19:26.924839 8426 leader_election.cc:288] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:26.926122 30357 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:26.926566 30358 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:26.926697 30358 leader_election.cc:302] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:26.927117 8426 raft_consensus.cc:2726] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:26.927580 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.941639 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.962574 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.962991 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.973738 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.974694 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:26.977914 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.010908 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.027447 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.033644 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:27.039650 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.075414 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.110282 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:27.142628 27698 maintenance_manager.cc:419] P bf8ac71b15f740a0a928911360773442: Scheduling FlushMRSOp(7c233aeb1e58418ba7b01f48917fabad): perf score=0.033351
W20240502 08:19:27.155766 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:19:27.181030 27588 maintenance_manager.cc:643] P bf8ac71b15f740a0a928911360773442: FlushMRSOp(7c233aeb1e58418ba7b01f48917fabad) complete. Timing: real 0.038s user 0.026s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":329,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1563,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":829,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6611,"threads_started":2,"wal-append.queue_time_us":6256}
I20240502 08:19:27.182680 27698 maintenance_manager.cc:419] P bf8ac71b15f740a0a928911360773442: Scheduling UndoDeltaBlockGCOp(7c233aeb1e58418ba7b01f48917fabad): 4586 bytes on disk
I20240502 08:19:27.183142 27588 maintenance_manager.cc:643] P bf8ac71b15f740a0a928911360773442: UndoDeltaBlockGCOp(7c233aeb1e58418ba7b01f48917fabad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
W20240502 08:19:27.205909 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.237071 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.299983 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:19:27.304538 27304 maintenance_manager.cc:419] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Scheduling FlushMRSOp(dd887cbb35c74e54a43d3dc6a2dde817): perf score=0.033394
W20240502 08:19:27.310457 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:27.326740 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:19:27.356726 27195 maintenance_manager.cc:643] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: FlushMRSOp(dd887cbb35c74e54a43d3dc6a2dde817) complete. Timing: real 0.052s user 0.028s sys 0.021s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":317,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1387,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":553,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":3213,"threads_started":2,"wal-append.queue_time_us":3084}
I20240502 08:19:27.359508 27304 maintenance_manager.cc:419] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Scheduling UndoDeltaBlockGCOp(dd887cbb35c74e54a43d3dc6a2dde817): 4840 bytes on disk
I20240502 08:19:27.360014 27195 maintenance_manager.cc:643] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: UndoDeltaBlockGCOp(dd887cbb35c74e54a43d3dc6a2dde817) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
W20240502 08:19:27.369078 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:27.382477 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.450378 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.496156 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:27.497226 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:27.534310 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.580346 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.598165 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.631049 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.645228 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:27.660444 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.662127 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.732381 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:27.745048 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.749207 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:19:27.825258 8161 raft_consensus.cc:482] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:27.825439 8161 raft_consensus.cc:504] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:19:27.826159 8161 leader_election.cc:288] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:27.827143 27194 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:27.827525 27193 leader_election.cc:334] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:27.827630 27193 leader_election.cc:302] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:27.827862 8161 raft_consensus.cc:2726] T 5e0313e85d444e098bfa35be89a3ad97 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 08:19:27.837440 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.866746 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.877898 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.882963 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.924667 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:27.965451 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:28.026546 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:28.028046 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:28.032383 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:28.099344 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:28.100749 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:28.101954 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:28.104837 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:28.141382 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:28.151822 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:28.199489 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:28.205951 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:28.246932 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:19:28.247638 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:28.261354 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:28.265044 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:28.288973 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:28.334237 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:28.341486 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:28.382136 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:28.496937 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:28.513228 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:28.579293 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:28.580356 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:28.637729 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:28.707203 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:28.752705 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:28.753707 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:28.807827 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:28.821089 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:28.831878 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:28.835887 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:28.838634 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:28.891440 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:28.893930 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:28.905337 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:28.908613 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:28.908609 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:19:28.993969 7577 raft_consensus.cc:482] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:28.994153 7577 raft_consensus.cc:504] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:28.994832 7577 leader_election.cc:288] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:28.995946 29964 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:28.996178 29961 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:28.996271 29961 leader_election.cc:302] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:28.996474 7577 raft_consensus.cc:2726] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:29.001518 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.018280 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:29.065752 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.101894 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:29.111980 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.132350 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.176506 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.211817 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.216157 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.247362 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.266086 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.277531 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.277976 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.311866 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.321591 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:29.364777 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:29.425796 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.430871 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.463729 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.468302 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.497216 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.498906 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:29.503582 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.505417 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.545743 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:29.578625 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:29.584254 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:29.585911 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.607491 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.657989 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.693782 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.714519 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.735692 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:29.744238 27193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 166 similar messages]
W20240502 08:19:29.759152 30096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 212 similar messages]
W20240502 08:19:29.789234 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:29.789211 27584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240502 08:19:29.801498 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:29.804906 29963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240502 08:19:29.858436 30358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240502 08:19:29.885257 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:29.890964 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.919898 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.921772 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.963609 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:29.963919 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:19:29.984414 8688 raft_consensus.cc:482] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:29.984575 8688 raft_consensus.cc:504] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:19:29.985139 8688 leader_election.cc:288] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:29.986470 29963 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:29.986725 29961 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:29.986838 29961 leader_election.cc:302] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:29.987004 8688 raft_consensus.cc:2726] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:30.072216 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:30.127707 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:30.160485 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:30.175940 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:30.208096 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:30.232586 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:30.232918 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:30.244464 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:30.282186 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:30.291577 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:30.296620 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:30.297947 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:19:30.317066 8689 raft_consensus.cc:482] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:30.317202 8689 raft_consensus.cc:504] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:19:30.317833 8689 leader_election.cc:288] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:30.319176 30355 leader_election.cc:334] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:30.319362 30357 leader_election.cc:334] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:30.319492 30357 leader_election.cc:302] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:30.319715 8689 raft_consensus.cc:2726] T dd644c6742854b79a212d858fa9df147 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:19:30.325335 8773 raft_consensus.cc:482] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:19:30.325461 8773 raft_consensus.cc:504] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } }
I20240502 08:19:30.325984 8773 leader_election.cc:288] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:30.327248 29963 leader_election.cc:334] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:30.327530 29964 leader_election.cc:334] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:30.327653 29964 leader_election.cc:302] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:30.327872 8773 raft_consensus.cc:2726] T 47b5771de7034e10acf2f40d0ac3c4d3 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:30.341450 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:30.369901 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:30.375984 8223 raft_consensus.cc:482] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:30.376199 8223 raft_consensus.cc:504] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } attrs { promote: false } }
I20240502 08:19:30.377048 8223 leader_election.cc:288] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:30.378567 27191 leader_election.cc:334] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:30.379050 27193 leader_election.cc:334] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:30.379199 27193 leader_election.cc:302] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:30.379472 8223 raft_consensus.cc:2726] T 2066c7aa052342a1aca3eb849aa251fc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:30.384318 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:19:30.387315 27304 maintenance_manager.cc:419] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Scheduling FlushMRSOp(bd640e48563f458cb2d56c94ed1e0347): perf score=0.033345
W20240502 08:19:30.421491 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:19:30.446435 27195 maintenance_manager.cc:643] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: FlushMRSOp(bd640e48563f458cb2d56c94ed1e0347) complete. Timing: real 0.059s user 0.035s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":362,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":1412,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":776,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":601,"threads_started":2,"wal-append.queue_time_us":448}
I20240502 08:19:30.448936 27304 maintenance_manager.cc:419] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Scheduling UndoDeltaBlockGCOp(bd640e48563f458cb2d56c94ed1e0347): 4925 bytes on disk
I20240502 08:19:30.449410 27195 maintenance_manager.cc:643] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: UndoDeltaBlockGCOp(bd640e48563f458cb2d56c94ed1e0347) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 08:19:30.454444 27698 maintenance_manager.cc:419] P bf8ac71b15f740a0a928911360773442: Scheduling FlushMRSOp(bd640e48563f458cb2d56c94ed1e0347): perf score=0.033367
W20240502 08:19:30.471518 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:19:30.481951 8816 raft_consensus.cc:482] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:30.482136 8816 raft_consensus.cc:504] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:19:30.482733 8816 leader_election.cc:288] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:30.484200 29961 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:30.484768 29964 leader_election.cc:334] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:30.484865 29964 leader_election.cc:302] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:30.485049 8816 raft_consensus.cc:2726] T ecc002b16f4642159e199298acd38392 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:19:30.486558 27588 maintenance_manager.cc:643] P bf8ac71b15f740a0a928911360773442: FlushMRSOp(bd640e48563f458cb2d56c94ed1e0347) complete. Timing: real 0.032s user 0.022s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":266,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1346,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":546,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":713,"threads_started":2,"wal-append.queue_time_us":623}
I20240502 08:19:30.488966 27698 maintenance_manager.cc:419] P bf8ac71b15f740a0a928911360773442: Scheduling UndoDeltaBlockGCOp(bd640e48563f458cb2d56c94ed1e0347): 4925 bytes on disk
I20240502 08:19:30.489677 27588 maintenance_manager.cc:643] P bf8ac71b15f740a0a928911360773442: UndoDeltaBlockGCOp(bd640e48563f458cb2d56c94ed1e0347) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 08:19:30.491005 27698 maintenance_manager.cc:419] P bf8ac71b15f740a0a928911360773442: Scheduling FlushMRSOp(c2776f857a5f47e9b8022639dc54757d): perf score=0.033373
W20240502 08:19:30.517103 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:19:30.521761 27588 maintenance_manager.cc:643] P bf8ac71b15f740a0a928911360773442: FlushMRSOp(c2776f857a5f47e9b8022639dc54757d) complete. Timing: real 0.031s user 0.025s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1354,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":502,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":220,"threads_started":1,"wal-append.queue_time_us":316}
I20240502 08:19:30.523422 27698 maintenance_manager.cc:419] P bf8ac71b15f740a0a928911360773442: Scheduling UndoDeltaBlockGCOp(c2776f857a5f47e9b8022639dc54757d): 4586 bytes on disk
I20240502 08:19:30.523897 27588 maintenance_manager.cc:643] P bf8ac71b15f740a0a928911360773442: UndoDeltaBlockGCOp(c2776f857a5f47e9b8022639dc54757d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
W20240502 08:19:30.562244 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:30.564545 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:30.579305 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:30.591349 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:30.604290 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:30.636608 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:30.697779 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:30.703226 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:30.772393 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:30.797374 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:30.800994 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:19:30.804558 8601 raft_consensus.cc:482] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:19:30.804682 8601 raft_consensus.cc:504] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:19:30.805176 8601 leader_election.cc:288] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:30.806421 27586 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:30.806649 27587 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:30.806735 27587 leader_election.cc:302] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:30.806911 8601 raft_consensus.cc:2726] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:30.814532 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:30.815150 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:30.830787 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:30.851392 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:30.897879 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:30.914383 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:19:30.952840 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:30.993556 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:31.058359 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.061017 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.075229 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.103284 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:31.203155 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:19:31.204910 8601 raft_consensus.cc:482] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:31.205032 8601 raft_consensus.cc:504] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { replace: true } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: NON_VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: true } }
I20240502 08:19:31.205639 8601 leader_election.cc:288] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:31.206873 27584 leader_election.cc:334] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:31.206998 27586 leader_election.cc:334] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:31.207087 27586 leader_election.cc:302] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:31.207279 8601 raft_consensus.cc:2726] T f1f416f090284b8d92d53662cc7a3b97 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:31.211378 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.246311 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:31.250602 8751 raft_consensus.cc:482] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:19:31.250739 8751 raft_consensus.cc:504] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "31a1e33dcdbf42f58ea72ce1c0afb0ad" member_type: VOTER last_known_addr { host: "127.20.207.8" port: 41619 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:19:31.251295 8751 leader_election.cc:288] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:31.252509 30357 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:31.252736 30358 leader_election.cc:334] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:31.252842 30358 leader_election.cc:302] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a1e33dcdbf42f58ea72ce1c0afb0ad; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:31.253170 8751 raft_consensus.cc:2726] T 8d309785afd8467dab3418bb63c95030 P 31a1e33dcdbf42f58ea72ce1c0afb0ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:31.266798 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:31.275352 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:31.323027 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.388254 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.471877 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:31.481145 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:31.522509 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.540741 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:31.543524 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.545220 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.556777 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.610905 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.610901 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:31.623545 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.631614 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:31.647235 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.659829 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.676155 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:19:31.689154 8601 raft_consensus.cc:482] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:31.689286 8601 raft_consensus.cc:504] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:19:31.689889 8601 leader_election.cc:288] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:31.691129 27584 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:31.691318 27587 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:31.691416 27587 leader_election.cc:302] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:31.691614 8601 raft_consensus.cc:2726] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:19:31.723877 8601 raft_consensus.cc:482] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:31.724020 8601 raft_consensus.cc:504] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } attrs { promote: false } }
I20240502 08:19:31.724577 8601 leader_election.cc:288] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:31.725844 27586 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:31.727244 27584 leader_election.cc:334] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:31.727356 27584 leader_election.cc:302] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:31.727552 8601 raft_consensus.cc:2726] T b1008f7127704fe8b3eb6f9246b0a79c P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:31.740370 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.747196 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.748813 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.790735 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.793753 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.797497 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:31.830227 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:31.866791 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.907912 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:31.987272 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:19:31.990445 8933 raft_consensus.cc:482] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:31.990622 8933 raft_consensus.cc:504] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
W20240502 08:19:31.990985 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:19:31.991189 8933 leader_election.cc:288] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:31.992254 27587 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:31.992362 27584 leader_election.cc:334] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:31.992480 27584 leader_election.cc:302] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:31.992677 8933 raft_consensus.cc:2726] T a6dce110666a4a038f5f46d48a3ec7af P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:32.018661 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:32.031589 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.032908 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:32.118556 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.167657 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:32.195003 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:32.200320 8932 raft_consensus.cc:482] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:32.200459 8932 raft_consensus.cc:504] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:19:32.201071 8932 leader_election.cc:288] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:32.202663 29964 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:32.202811 29963 leader_election.cc:334] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:32.202896 29963 leader_election.cc:302] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:32.203110 8932 raft_consensus.cc:2726] T 0bea60d096f9408c9f4a857f8824d6a2 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:32.208317 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.213799 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.236495 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.278841 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:32.298880 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.304502 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.307556 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.347975 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:32.380049 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:32.389479 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.438278 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:32.447054 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.549427 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:19:32.552266 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.605645 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:32.623221 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.628260 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.675154 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.684125 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.690232 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.725405 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.770707 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.788998 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:32.796221 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:32.817494 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.818379 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.839547 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.889242 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:32.908551 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.913049 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:32.960533 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:33.027302 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:33.091883 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:33.093839 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:33.132164 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:33.146368 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:33.158211 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:33.189270 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:33.213263 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:33.224334 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:33.267843 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:33.280253 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:33.289142 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:33.292477 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:33.294673 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:33.306860 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:33.363572 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:33.455649 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:19:33.470738 27304 maintenance_manager.cc:419] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Scheduling FlushMRSOp(6978aa3955504c548778066ddd0910e5): perf score=0.033336
W20240502 08:19:33.479825 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:33.482018 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:19:33.502089 27195 maintenance_manager.cc:643] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: FlushMRSOp(6978aa3955504c548778066ddd0910e5) complete. Timing: real 0.031s user 0.016s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":260,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":1543,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":593,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":340,"threads_started":2,"wal-append.queue_time_us":245}
W20240502 08:19:33.502794 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:33.504007 27304 maintenance_manager.cc:419] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Scheduling UndoDeltaBlockGCOp(6978aa3955504c548778066ddd0910e5): 4586 bytes on disk
I20240502 08:19:33.504395 27195 maintenance_manager.cc:643] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: UndoDeltaBlockGCOp(6978aa3955504c548778066ddd0910e5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240502 08:19:33.540789 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:19:33.544704 27698 maintenance_manager.cc:419] P bf8ac71b15f740a0a928911360773442: Scheduling FlushMRSOp(b82841ff10f24cf7a0cf490be3defe69): perf score=0.033364
W20240502 08:19:33.556797 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:33.575821 27588 maintenance_manager.cc:643] P bf8ac71b15f740a0a928911360773442: FlushMRSOp(b82841ff10f24cf7a0cf490be3defe69) complete. Timing: real 0.031s user 0.025s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":323,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1327,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":575,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":401,"threads_started":2,"wal-append.queue_time_us":261}
I20240502 08:19:33.577880 27698 maintenance_manager.cc:419] P bf8ac71b15f740a0a928911360773442: Scheduling UndoDeltaBlockGCOp(b82841ff10f24cf7a0cf490be3defe69): 4840 bytes on disk
I20240502 08:19:33.578392 27588 maintenance_manager.cc:643] P bf8ac71b15f740a0a928911360773442: UndoDeltaBlockGCOp(b82841ff10f24cf7a0cf490be3defe69) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240502 08:19:33.579720 27698 maintenance_manager.cc:419] P bf8ac71b15f740a0a928911360773442: Scheduling FlushMRSOp(6978aa3955504c548778066ddd0910e5): perf score=0.033366
I20240502 08:19:33.609938 27588 maintenance_manager.cc:643] P bf8ac71b15f740a0a928911360773442: FlushMRSOp(6978aa3955504c548778066ddd0910e5) complete. Timing: real 0.030s user 0.015s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1224,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":560,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":285,"threads_started":1,"wal-append.queue_time_us":366}
I20240502 08:19:33.612226 27698 maintenance_manager.cc:419] P bf8ac71b15f740a0a928911360773442: Scheduling UndoDeltaBlockGCOp(6978aa3955504c548778066ddd0910e5): 4586 bytes on disk
I20240502 08:19:33.612783 27588 maintenance_manager.cc:643] P bf8ac71b15f740a0a928911360773442: UndoDeltaBlockGCOp(6978aa3955504c548778066ddd0910e5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240502 08:19:33.644532 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:33.680089 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:33.752141 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:33.796622 8601 raft_consensus.cc:482] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:19:33.796798 8601 raft_consensus.cc:504] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } }
I20240502 08:19:33.797402 8601 leader_election.cc:288] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:33.799005 27587 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:33.799139 27586 leader_election.cc:334] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:33.799242 27586 leader_election.cc:302] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:33.799440 8601 raft_consensus.cc:2726] T fee0386181d84b18803effc8af1bbbc4 P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:33.806958 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:33.827529 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:33.854537 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:33.861786 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:33.865974 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:33.877034 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:33.880070 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:33.938059 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:33.952876 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:33.970824 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.005263 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.016067 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.052843 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.057076 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:34.091584 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:34.113590 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:34.159422 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:34.191735 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.264760 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.271445 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.279510 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.292737 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.292783 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:34.321643 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.327922 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:19:34.350358 8601 raft_consensus.cc:482] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:34.350499 8601 raft_consensus.cc:504] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:34.351013 8601 leader_election.cc:288] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:34.352442 27584 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:34.362890 27587 leader_election.cc:334] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:34.363086 27587 leader_election.cc:302] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:34.363396 8601 raft_consensus.cc:2726] T 7c233aeb1e58418ba7b01f48917fabad P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:19:34.398278 9084 raft_consensus.cc:482] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83e171a5e16c4008923f1d43fc5ce7b1)
I20240502 08:19:34.398480 9084 raft_consensus.cc:504] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } attrs { promote: false } }
I20240502 08:19:34.399259 9084 leader_election.cc:288] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:34.403280 27193 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:34.408861 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:34.411054 27194 leader_election.cc:334] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:34.411170 27194 leader_election.cc:302] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:34.411389 9084 raft_consensus.cc:2726] T 0ed6b8adc62f471696cbf0330524e353 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:34.421154 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.435479 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.451138 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.461962 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.558501 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.595178 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.645066 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:34.676177 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:34.686553 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:34.702071 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:34.712322 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.751032 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.752534 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.753546 27193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 171 similar messages]
W20240502 08:19:34.754182 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.760191 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:34.772573 30095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 210 similar messages]
W20240502 08:19:34.777801 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:34.797901 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.798172 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:34.807045 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.814165 27587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111) [suppressed 176 similar messages]
I20240502 08:19:34.862159 9063 raft_consensus.cc:482] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:34.862316 9063 raft_consensus.cc:504] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "84cda20738d2452da166e31fc6da57a6" member_type: VOTER last_known_addr { host: "127.20.207.6" port: 38899 } attrs { promote: false } }
I20240502 08:19:34.862939 9063 leader_election.cc:288] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:34.864223 30095 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:34.864356 30096 leader_election.cc:334] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:34.864442 30096 leader_election.cc:302] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cda20738d2452da166e31fc6da57a6; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:34.864660 9063 raft_consensus.cc:2726] T db55b42c724246da8d9cb23369826279 P 84cda20738d2452da166e31fc6da57a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:34.870651 30355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240502 08:19:34.871345 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.881341 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.890810 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.938966 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:34.940858 29963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 08:19:34.951025 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:34.970281 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.000437 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:35.002511 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:35.054147 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.078909 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:35.087512 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.122256 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.160137 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.189296 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:35.198101 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.231266 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.251542 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:35.305681 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.320525 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.360651 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.397881 27103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:35.399497 27103 catalog_manager.cc:4573] TS b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6cf739b314d84db08bc4cb32b0e5b0f4: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:35.400372 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.403584 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.448760 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.548288 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.566262 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:35.567114 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.575601 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.576481 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.606057 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:19:35.628453 27698 maintenance_manager.cc:419] P bf8ac71b15f740a0a928911360773442: Scheduling FlushMRSOp(33aee3df5eb242ef9cc056d25f6d6a82): perf score=0.033356
W20240502 08:19:35.670128 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:35.679752 27588 maintenance_manager.cc:643] P bf8ac71b15f740a0a928911360773442: FlushMRSOp(33aee3df5eb242ef9cc056d25f6d6a82) complete. Timing: real 0.051s user 0.023s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":10452,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":1543,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":905,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":9977,"threads_started":2,"wal-append.queue_time_us":375}
I20240502 08:19:35.682246 27698 maintenance_manager.cc:419] P bf8ac71b15f740a0a928911360773442: Scheduling UndoDeltaBlockGCOp(33aee3df5eb242ef9cc056d25f6d6a82): 4586 bytes on disk
I20240502 08:19:35.682762 27588 maintenance_manager.cc:643] P bf8ac71b15f740a0a928911360773442: UndoDeltaBlockGCOp(33aee3df5eb242ef9cc056d25f6d6a82) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
W20240502 08:19:35.698894 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:19:35.770601 27304 maintenance_manager.cc:419] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Scheduling FlushMRSOp(09aa1b5d99114b3095be08c9547fbb54): perf score=0.033395
W20240502 08:19:35.771590 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:35.778489 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.792191 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:35.793813 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:35.798492 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.809217 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:35.810328 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:19:35.819303 27195 maintenance_manager.cc:643] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: FlushMRSOp(09aa1b5d99114b3095be08c9547fbb54) complete. Timing: real 0.048s user 0.034s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":605,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":1336,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":706,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":777,"threads_started":2,"wal-append.queue_time_us":338}
I20240502 08:19:35.821240 27304 maintenance_manager.cc:419] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Scheduling UndoDeltaBlockGCOp(09aa1b5d99114b3095be08c9547fbb54): 4840 bytes on disk
I20240502 08:19:35.821647 27195 maintenance_manager.cc:643] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: UndoDeltaBlockGCOp(09aa1b5d99114b3095be08c9547fbb54) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
W20240502 08:19:35.842043 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:36.002055 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:19:36.013370 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:19:36.014133 8601 raft_consensus.cc:482] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:36.014269 8601 raft_consensus.cc:504] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf8ac71b15f740a0a928911360773442" member_type: VOTER last_known_addr { host: "127.20.207.4" port: 33309 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } attrs { promote: false } }
I20240502 08:19:36.014848 8601 leader_election.cc:288] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469)
W20240502 08:19:36.015973 27584 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:36.016466 27587 leader_election.cc:334] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:36.016574 27587 leader_election.cc:302] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf8ac71b15f740a0a928911360773442; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:36.016777 8601 raft_consensus.cc:2726] T ca8d1416a54344feae081ed7b19dd1de P bf8ac71b15f740a0a928911360773442 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:36.055442 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:36.058938 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:36.108641 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.117110 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.147863 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:36.200798 9197 raft_consensus.cc:482] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:36.200958 9197 raft_consensus.cc:504] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:19:36.201738 9197 leader_election.cc:288] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:36.203327 29961 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:36.204025 29964 leader_election.cc:334] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
I20240502 08:19:36.204130 29964 leader_election.cc:302] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:36.204361 9197 raft_consensus.cc:2726] T 5a254106456240879784a4d85d56fca5 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:36.245221 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.295974 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:36.297041 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:36.316499 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.353500 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.388044 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.404381 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.429854 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.487125 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:36.511638 30096 consensus_peers.cc:485] T 4165b86bc05e47f6a9803fb45ca01569 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.531642 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.532925 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:36.561136 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.588023 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.629690 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.639070 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.663477 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:36.670771 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.673163 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:36.690532 27587 consensus_peers.cc:485] T 56cb28bc9e264d4882d482499b950349 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:36.702199 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:36.710625 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.710824 27194 consensus_peers.cc:485] T dd887cbb35c74e54a43d3dc6a2dde817 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:19:36.719295 9197 raft_consensus.cc:482] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49c5c8f70e494edd9fcda40cc0af6cc4)
I20240502 08:19:36.719429 9197 raft_consensus.cc:504] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } }
I20240502 08:19:36.719976 9197 leader_election.cc:288] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:36.721364 29963 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:36.721767 29961 leader_election.cc:334] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:36.721879 29961 leader_election.cc:302] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, 83e171a5e16c4008923f1d43fc5ce7b1
I20240502 08:19:36.722077 9197 raft_consensus.cc:2726] T 556fc9ebf7bd4b859f584d46411c2403 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:36.761890 29961 consensus_peers.cc:485] T 09aa1b5d99114b3095be08c9547fbb54 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.797626 30095 consensus_peers.cc:485] T 04a0a58eb52f41bf818e5102996d09e0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:36.870537 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:36.901798 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:36.963263 30093 consensus_peers.cc:485] T eca2618572fe4900bd420bb59c22381f P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.983736 29961 consensus_peers.cc:485] T c3a55662c6e346f0b73df334be4b14e3 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:36.988459 27193 consensus_peers.cc:485] T a9e5e67e3fe54491841b33d0d4bd047b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.019776 27194 consensus_peers.cc:485] T feb9cd278950480fb63be0be950d190b P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:19:37.035192 9379 raft_consensus.cc:482] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:37.035342 9379 raft_consensus.cc:504] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:37.036036 9379 leader_election.cc:288] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:37.037067 29964 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:37.037266 29961 leader_election.cc:334] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:37.037364 29961 leader_election.cc:302] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:37.037595 9379 raft_consensus.cc:2726] T e80b9a98bdd041b1ac97af055951a92a P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:37.065990 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:37.145319 30357 consensus_peers.cc:485] T 168e71e71b0340fd953d5644b8b8c028 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.146266 30355 consensus_peers.cc:485] T f5a0e83549644add82e39c5720805b95 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.153254 30355 consensus_peers.cc:485] T 15a333c65455473fbdc26d1ec137aebc P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.160991 30357 consensus_peers.cc:485] T e5af418a0ae043a387b5030a00b47e91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:37.191318 30093 consensus_peers.cc:485] T b8b7b807126c40939ef272fd355b54e4 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:19:37.196269 9197 raft_consensus.cc:482] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:37.196409 9197 raft_consensus.cc:504] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "83e171a5e16c4008923f1d43fc5ce7b1" member_type: VOTER last_known_addr { host: "127.20.207.2" port: 46291 } } peers { permanent_uuid: "1f0bf4bc3c5146539146b825b048bff3" member_type: VOTER last_known_addr { host: "127.20.207.5" port: 37815 } attrs { promote: false } }
I20240502 08:19:37.197062 9197 leader_election.cc:288] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291)
W20240502 08:19:37.198199 29964 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:37.198334 29963 leader_election.cc:334] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
I20240502 08:19:37.198441 29963 leader_election.cc:302] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f0bf4bc3c5146539146b825b048bff3; no voters: 83e171a5e16c4008923f1d43fc5ce7b1, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:37.198626 9197 raft_consensus.cc:2726] T f2b08a6b38f945b1b99f1d8ad1fd0ae0 P 1f0bf4bc3c5146539146b825b048bff3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:37.204401 30095 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.216595 27584 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:37.222084 27191 consensus_peers.cc:485] T 08245cda17c44a77bcde2f864d2045ec P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:37.236249 27193 consensus_peers.cc:485] T 1662e908146944fd8e74d901e497e35d P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:37.238667 30358 consensus_peers.cc:485] T 1adfeb89dc5341a2b8cf962eb28da45e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.244724 30095 consensus_peers.cc:485] T 9edac369e0ae4c3c990bbb8dd845b27c P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:37.252733 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:37.270071 30095 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.314615 30096 consensus_peers.cc:485] T 16b897307b4c4cbea72f814f64f1e7d7 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.343730 30093 consensus_peers.cc:485] T b9d6207b6bb3436f996a069ff910a56e P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.375010 27194 consensus_peers.cc:485] T e65281f04bee4c299aa019adb454bbf5 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.391521 30096 consensus_peers.cc:485] T d122f68d36024614a017e8cc3a1224c1 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.400914 29963 consensus_peers.cc:485] T 70f4c40bc22947468dfd15804a4e66cd P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.432749 27193 consensus_peers.cc:485] T a3724af3b82c43b0a1036f4d9703a3a9 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.440150 29961 consensus_peers.cc:485] T 41f6a5115030489fac2e1edce67a4b52 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.440814 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:37.581823 30093 consensus_peers.cc:485] T 57505255575249e698bf35929619423d P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.600253 30358 consensus_peers.cc:485] T e7064865693149b980ba54fb8e65bc48 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:37.638602 27191 consensus_peers.cc:485] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.664348 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:37.683482 30357 consensus_peers.cc:485] T 2e9b191ce9234c028b990efd74a59e2e P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:37.686882 27584 consensus_peers.cc:485] T d9f41c94eaf04ac5ad77d3124f6368c9 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.699282 29963 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.708875 30093 consensus_peers.cc:485] T 757f65cd77bf49d882c2545673282e03 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.784727 27586 consensus_peers.cc:485] T cad036d297bd45d7b7c5a1f19356906f P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.789111 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:37.790808 27584 consensus_peers.cc:485] T c2776f857a5f47e9b8022639dc54757d P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.811028 30095 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.821363 30095 consensus_peers.cc:485] T 51201cc579394a2b819278a5943485a0 P 84cda20738d2452da166e31fc6da57a6 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:19:37.836181 27304 maintenance_manager.cc:419] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Scheduling FlushMRSOp(fef4f56761bb458c8d84807feb38a617): perf score=0.033365
W20240502 08:19:37.863878 30358 consensus_peers.cc:485] T 4c78e068ec3a451c892d58174b3f837c P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:37.864290 30357 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:19:37.869623 27195 maintenance_manager.cc:643] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: FlushMRSOp(fef4f56761bb458c8d84807feb38a617) complete. Timing: real 0.033s user 0.023s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":295,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1400,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":638,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":390,"threads_started":2,"wal-append.queue_time_us":240}
I20240502 08:19:37.871243 27304 maintenance_manager.cc:419] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: Scheduling UndoDeltaBlockGCOp(fef4f56761bb458c8d84807feb38a617): 4840 bytes on disk
I20240502 08:19:37.871659 27195 maintenance_manager.cc:643] P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60: UndoDeltaBlockGCOp(fef4f56761bb458c8d84807feb38a617) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240502 08:19:37.888707 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:37.903609 30355 consensus_peers.cc:485] T d3a3473a93f14baaac48b9b7f47ac2a1 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:37.909499 29961 consensus_peers.cc:485] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:19:37.950544 27698 maintenance_manager.cc:419] P bf8ac71b15f740a0a928911360773442: Scheduling FlushMRSOp(cad036d297bd45d7b7c5a1f19356906f): perf score=0.033392
I20240502 08:19:37.980386 27588 maintenance_manager.cc:643] P bf8ac71b15f740a0a928911360773442: FlushMRSOp(cad036d297bd45d7b7c5a1f19356906f) complete. Timing: real 0.030s user 0.023s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":335,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1401,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":547,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":443,"threads_started":2,"wal-append.queue_time_us":321}
I20240502 08:19:37.982286 27698 maintenance_manager.cc:419] P bf8ac71b15f740a0a928911360773442: Scheduling UndoDeltaBlockGCOp(cad036d297bd45d7b7c5a1f19356906f): 4586 bytes on disk
I20240502 08:19:37.982771 27588 maintenance_manager.cc:643] P bf8ac71b15f740a0a928911360773442: UndoDeltaBlockGCOp(cad036d297bd45d7b7c5a1f19356906f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
W20240502 08:19:38.048926 27194 consensus_peers.cc:485] T af2f11471c064f48924191fc2607d31c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:38.073143 27586 consensus_peers.cc:485] T 459e9f5bf6d24ba9b36686d49ff12e2b P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:38.074483 29961 consensus_peers.cc:485] T 1edc0a17b5f2466cbe2cf5a140689734 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:38.105691 27191 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:38.151149 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:38.193367 27587 consensus_peers.cc:485] T de4b822c4aca4f7f92b73cc9279ddec5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:38.226989 27587 consensus_peers.cc:485] T b82841ff10f24cf7a0cf490be3defe69 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:38.231032 27587 consensus_peers.cc:485] T 2abc11a238a542e4982452b5ce341101 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:38.233990 27191 consensus_peers.cc:485] T 903502e57ab2430383807b0d46a39847 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:38.294122 27193 consensus_peers.cc:485] T 81e73229c0d145e7bb81761bf4b820dc P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:38.307857 30096 consensus_peers.cc:485] T eb5345b931504cfdbb272189b719efa4 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:38.339637 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:38.347635 27584 consensus_peers.cc:485] T 56542089a8e14afb9ba3d9e7cbbbcdf6 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:38.354187 30357 consensus_peers.cc:485] T 201d79895c2741dd9d9de0cc9f605a91 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:38.375445 27194 consensus_peers.cc:485] T e67c99e57a73443da13527ed33326a4e P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:38.391654 27191 consensus_peers.cc:485] T 2dfa509d47e54b0f9ccacc9954981fb6 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:38.512729 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:38.575160 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:38.581382 30096 consensus_peers.cc:485] T 0d33916984034ab99f957026b3e30d38 P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:38.602051 30093 consensus_peers.cc:485] T 65ac899c820a43e4ab57454e224edc83 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:38.608155 30096 consensus_peers.cc:485] T a52fe5caeb364b868a1965633a7da09e P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:38.638615 27191 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:38.682248 27584 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:38.717262 29963 consensus_peers.cc:485] T 6cf739b314d84db08bc4cb32b0e5b0f4 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:19:38.718233 30357 consensus_peers.cc:485] T bc8fe72edcef41a28fdeea86d501f93b P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:38.727777 29964 consensus_peers.cc:462] T cef42b07900c4b4e8589f284c011c300 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:38.775226 30096 consensus_peers.cc:485] T 0a5a0175799e478c9f53ecc82ba090ae P 84cda20738d2452da166e31fc6da57a6 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:38.839673 27586 consensus_peers.cc:485] T 84953bd8daba4885be1f5f9ad4149aba P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:19:38.881042 9512 raft_consensus.cc:482] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b01e7c8a302d41078fa39cfef665ff59)
I20240502 08:19:38.881228 9512 raft_consensus.cc:504] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c0e4a6cec3c4ebb92a5abf1d8b8ba60" member_type: VOTER last_known_addr { host: "127.20.207.1" port: 38505 } } peers { permanent_uuid: "b01e7c8a302d41078fa39cfef665ff59" member_type: VOTER last_known_addr { host: "127.20.207.3" port: 44469 } } peers { permanent_uuid: "49c5c8f70e494edd9fcda40cc0af6cc4" member_type: VOTER last_known_addr { host: "127.20.207.7" port: 40343 } attrs { promote: false } }
I20240502 08:19:38.881846 9512 leader_election.cc:288] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469), 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343)
W20240502 08:19:38.883445 27194 leader_election.cc:334] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111)
W20240502 08:19:38.883664 27191 leader_election.cc:334] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
I20240502 08:19:38.902405 27191 leader_election.cc:302] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0e4a6cec3c4ebb92a5abf1d8b8ba60; no voters: 49c5c8f70e494edd9fcda40cc0af6cc4, b01e7c8a302d41078fa39cfef665ff59
I20240502 08:19:38.902667 9512 raft_consensus.cc:2726] T fef4f56761bb458c8d84807feb38a617 P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:38.950542 29964 consensus_peers.cc:485] T 387e9ef5899541b0be27a41dbcf4475c P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:38.981489 27193 consensus_peers.cc:462] T 3bfa383343184f18969fa52ed97ab07c P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111)
W20240502 08:19:38.988910 30357 consensus_peers.cc:485] T 95fc757372c84ca9928f9ae8133b33f7 P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:38.988943 27584 consensus_peers.cc:485] T ec6c977cac054b758a43d846e821707a P bf8ac71b15f740a0a928911360773442 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:39.026779 27587 consensus_peers.cc:485] T fefcc470084f4ec9be6ba45d3ba22ce5 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:39.049927 27194 consensus_peers.cc:485] T 3047a6db2c9d438eb5720d848b9ea5bd P 1c0e4a6cec3c4ebb92a5abf1d8b8ba60 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:39.050645 30355 consensus_peers.cc:485] T 73548c1dd45f4c31bf3caf89b6d9ebac P 31a1e33dcdbf42f58ea72ce1c0afb0ad -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:39.055586 29963 consensus_peers.cc:485] T 26545a868bd6415c9c9db4869b2841c7 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:39.139358 30093 consensus_peers.cc:462] T c30720c1b245438db3eb4bea2b8bd930 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111)
W20240502 08:19:39.146919 30093 consensus_peers.cc:485] T 0524d946c6984ebcbf9bbd28d90c3844 P 84cda20738d2452da166e31fc6da57a6 -> Peer 49c5c8f70e494edd9fcda40cc0af6cc4 (127.20.207.7:40343): Couldn't send request to peer 49c5c8f70e494edd9fcda40cc0af6cc4. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.7:40343: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:39.165295 29964 consensus_peers.cc:485] T d00f25f29f5d4db2a2261dd37c0793fa P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:19:39.175699 27586 consensus_peers.cc:485] T b6a26e01d42b429694641f0ceca11265 P bf8ac71b15f740a0a928911360773442 -> Peer 83e171a5e16c4008923f1d43fc5ce7b1 (127.20.207.2:46291): Couldn't send request to peer 83e171a5e16c4008923f1d43fc5ce7b1. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.2:46291: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:39.188629 29964 consensus_peers.cc:485] T e2b9488fe8ac46099b7e8adf4d0b9cd5 P 1f0bf4bc3c5146539146b825b048bff3 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn't send request to peer b01e7c8a302d41078fa39cfef665ff59. Status: Network error: Client connection negotiation failed: client connection to 127.20.207.3:44469: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:19:39.200992 27587 consensus_peers.cc:485] T 56d2a9f088564a66bf6662a4ddbb8145 P bf8ac71b15f740a0a928911360773442 -> Peer b01e7c8a302d41078fa39cfef665ff59 (127.20.207.3:44469): Couldn'